19

Some people even claim that Google result counts are a meaningless metric.

Well, “meaningless” might be an overstatement, but as a matter of fact, Google counts can be incorrect by the order of several digits or more. I think that many people trust Google counts without knowing their inaccuracy. Please think twice before using Google counts as an indicator of how common a certain expression is.

This confused some people on our site:

There is a post on meta.english.stackexchange.com on the same topic.

Please try by yourself. You can see the inaccuracy of counts by yourself. Here is what I tried, reproducing the example in the second link above. I searched "られてられて" (with quotation marks) on Google. It said that there were about 37,800,000 results. (This number seems to vary from time to time.) We may take this large number as an evidence that the form “られてられて” is fairly common. However, when I went forward in the search results, the list came to an end after only 417 results! At the end of the list, Google showed:

In order to show you the most relevant results, we have omitted some entries very similar to the 417 already displayed.
If you like, you can repeat the search with the omitted results included.

I clicked the link to repeat the search with the omitted results included, and did the same thing. This time it showed 500 results, which is larger than 417 but not even close to 37,800,000. This means that there is some problem with Google counts, and we should not believe them without thinking.

8
  • 1
    Google-fundamentalists.
    – user458
    Oct 4, 2011 at 22:12
  • 1
    Because I was one of them before I learned about the inaccuracy, I cannot blame people who trust Google counts….
    – Tsuyoshi Ito
    Oct 4, 2011 at 22:41
  • The count may be tricky. But I am amazed at the fact that there are so many people who thinks it academic to just adopt the result from google without any consultation to informants or fieldwork, and calling it a "research".
    – user458
    Oct 4, 2011 at 22:50
  • 3
    Even if the results are accurate, they do next to nothing to prove whether something is right or wrong. The most one can deduce is "It is a commonly encountered term". Just because it has a high amount of hits does not make it right. Probabilistic reasoning is not a good enough justification for something.
    – Flaw Mod
    Oct 5, 2011 at 6:10
  • 1
    @sawa: I see. There is no way to stop hobbyists from using inaccurate methods, but I hope that professional linguists and students in linguistics are not like that….
    – Tsuyoshi Ito
    Oct 5, 2011 at 11:25
  • 2
    @Flaw Absolutely. google.com/search?q="maintainence" gives me 24.3m results. Even if we have a billion valid results, it just doesn't make "maintainence" the correct spelling for "maintenance"
    – Pacerier
    Oct 7, 2011 at 13:23
  • 5
    @Flaw That certainly depends on your definition of "right". One thing is what textbooks/language-governing bodies/prescriptive grammarians prescribe, another thing is the language which is actually being used. Most serious linguists are at least as interested in the latter.
    – dainichi
    May 7, 2012 at 1:22
  • I'd suggest an edit but Stack Exchange does not allow doing so. The FGA is now at jdebp.uk./FGA/…
    – JdeBP
    Jun 13, 2019 at 13:24

3 Answers 3

Reset to default
10

The Balanced Corpus of Contemporary Written Japanese (BCCWJ) is publicly available and can be searched for free using a site called 少納言. It consists of a hundred million words of published Japanese from 1971-2008. The result counts should be accurate, and in many situations it should be a better tool than Google for researching Japanese usage.

More information is available in both Japanese and English.

There's also a paper available with more details, titled "KOTONOHA and BCCWJ: Development of a Balanced Corpus of Contemporary Written Japanese" (PDF format).

I'm not sure if many people on JLU know that this resource exists, and I certainly haven't noticed many answers citing it. I hope people consider it and use it (or other corpora) when they feel it's appropriate to do so.

1
  • 1
    I had tried it before, but as you can see, I recently started using it to answer some of the questions on the site. Thank you for reminding me of it!
    – Tsuyoshi Ito
    Nov 4, 2013 at 1:58
4

For reference. http://searchengineland.com/why-google-cant-count-results-properly-53559

4
  • Even after reading this, I don't really understand how/why the estimates are so inaccurate...
    – atlantiza
    Jun 13, 2012 at 20:01
  • 1
    Can you summarize what the article says, in case it suffers link rot?
    – Andrew Grimm
    Jun 27, 2012 at 22:08
  • @AndrewGrimm I do not want to spend time doing it. If the link becomes dead, I will rather delete this answer.
    – user458
    Jun 28, 2012 at 1:31
  • To summarize briefly, computation time is valuable, so search engine algorithms try to perform the bare minimum amount of computation needed to deliver good results on the first page. If you signal that you want more in-depth results, such as by checking up to the 417th result, the algorithm is forced to do more computation, and only then does it give the better estimate.
    – HAL
    Mar 28, 2013 at 5:02
0

Thing is, Google only returns a maximum of 1000 results per search. If you click on this link1, you'll see the message

Sorry, Google does not serve more than 1000 results for any query. (You asked for results starting from 1000.)

Another way to test this is to first change your search settings to 100 results per page2 (you'll need to check "Never show Instant results"), then search for something common, e.g. "snail". You'll never get more than 10 pages.

On top of that, you get the "omission of similar entries". So you always get less than 1000 results.

That said, if you get very few results - say 50 or less - it might be an indicator that what you are searching might be wrong. But in terms of comparing multiple terms/phrases to see which is most common, comparing their Google hits (either the approximated number or the post-omission number) is not a reliable method. So searching corpora, like what snailplane suggested suggested is the way to go.

On a related note, folks at JMDict use Google N-grams, for example if you look at this edit history and this edit history. Found this Yahoo group message that confirms Jim Breen has access to a copy. Makes me wonder if any of Japanese SE users have access to that too.

1 Search query adapted from this Stack Overflow answer
2 The change to 100 results/page is so that you don't have to click through 100 pages.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .