Is there any open-source tokenizer of Japanese sentences? For example (according to this answer)


Becomes something like

住宅地域 に おける 本機 の 使用 は 有害な 電波妨害 を 引き起こす こと が あり、 その 場合 ユーザー は 自己負担 で 電波妨害 の 問題 を 解決 しなければなりません。
  • 1
    Mecab (code.google.com/p/mecab) should be able to do tokenization among other things.
    – undercat
    Oct 20, 2013 at 2:20
  • Note that the resources FAQ actually had several of the answers listed. I've added the two that weren't included.
    – jkerian
    Oct 22, 2013 at 6:10

2 Answers 2


[茶筅]{Chasen}、[案山子]{kakasi}、MeCab([和布蕪]{めかぶ}) are famous. For the details of each softwares, please Google them.



To cite their website:

Kuromoji supports standard morphological analysis features such as

  • Word segmentation - segmenting text into words (morphemes)
  • Part-of-speech tagging - assign word-categories (nouns, verbs, particles, adjectives, etc.)
  • Lemmatization - get dictionary forms for inflected verbs and adjectives Readings - extract readings for kanji

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .