I am a beginner and wonder if it would be helpful to identify the experience level of users in some way, before the questions are even asked. Have the level next to the user's name.
This is a classic example of what Stack Exchange refers to as a "meta tag" (not to be confused with a tag on the meta site):
The reason meta-tags are a problem is that they do not describe the content of the question. They describe some other aspect of the question, like the author’s skill level, or the author’s motivation for asking it, or generally what “kind” of question it is (poll, how-to, etc.).
Meta-tags are actually a subset of a larger problem that I usually call dependent tags. These are tags that don’t say anything by themselves – you can’t tell what the question is about unless they’re paired with some other tag (or several of them). These tags are a problem because people don’t realize this and will often use that as the question’s only tag.
As a general rule, we prefer not to have tags of this sort on any Stack Exchange site, and that includes Japanese.SE. For more information, please see the blog post linked above.