Is there any open-source tokenizer of Japanese sentences? For example (according to this answer)


Becomes something like

住宅地域 に おける 本機 の 使用 は 有害な 電波妨害 を 引き起こす こと が あり、 その 場合 ユーザー は 自己負担 で 電波妨害 の 問題 を 解決 しなければなりません。

migrated from Oct 20 '13 at 7:09

This question came from our site for students, teachers, and linguists wanting to discuss the finer points of the Japanese language.

Mecab ( should be able to do tokenization among other things. – vovick Oct 20 '13 at 2:20
Note that the resources FAQ actually had several of the answers listed. I've added the two that weren't included. – jkerian Oct 22 '13 at 6:10

2 Answers 2

up vote 10 down vote accepted

[茶筅]{Chasen}、[案山子]{kakasi}、MeCab([和布蕪]{めかぶ}) are famous. For the details of each softwares, please Google them.



To cite their website:

Kuromoji supports standard morphological analysis features such as

  • Word segmentation - segmenting text into words (morphemes)
  • Part-of-speech tagging - assign word-categories (nouns, verbs, particles, adjectives, etc.)
  • Lemmatization - get dictionary forms for inflected verbs and adjectives Readings - extract readings for kanji

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .