Is there any open-source tokenizer of Japanese sentences? For example (according to this answer)

住宅地域における本機の使用は有害な電波妨害を引き起こすことがあり、その場合ユーザーは自己負担で電波妨害の問題を解決しなければなりません。

Becomes something like

住宅地域 に おける 本機 の 使用 は 有害な 電波妨害 を 引き起こす こと が あり、 その 場合 ユーザー は 自己負担 で 電波妨害 の 問題 を 解決 しなければなりません。
share
1  
Mecab (code.google.com/p/mecab) should be able to do tokenization among other things. –  vovick Oct 20 '13 at 2:20
    
Note that the resources FAQ actually had several of the answers listed. I've added the two that weren't included. –  jkerian Oct 22 '13 at 6:10
add comment

migrated from japanese.stackexchange.com Oct 20 '13 at 7:09

This question came from our site for students, teachers, and linguists wanting to discuss the finer points of the Japanese language.

2 Answers

up vote 8 down vote accepted

[茶筅]{Chasen}、[案山子]{kakasi}、MeCab([和布蕪]{めかぶ}) are famous. For the details of each softwares, please Google them.

share
add comment

Kuromoji

To cite their website:

Kuromoji supports standard morphological analysis features such as

  • Word segmentation - segmenting text into words (morphemes)
  • Part-of-speech tagging - assign word-categories (nouns, verbs, particles, adjectives, etc.)
  • Lemmatization - get dictionary forms for inflected verbs and adjectives Readings - extract readings for kanji
share
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .