Japanese Word Segmentation


WordSegmenterModel (WSM) is based on maximum entropy probability model to detect word boundaries in Chinese text. Chinese text is written without white space between the words, and a computer-based application cannot know a priori which sequence of ideograms form a word. In many natural language processing tasks such as part-of-speech (POS) and named entity recognition (NER) require word segmentation as a initial step.


  • Xue, Nianwen. “Chinese word segmentation as character tagging.” International Journal of Computational Linguistics & Chinese Language Processing, Volume 8, Number 1, February 2003: Special Issue on Word Formation and Chinese Language Processing. 2003.).


How to use

Use as part of an nlp pipeline as a substitute of the Tokenizer stage.

word_segmenter = WordSegmenterModel.pretrained('wordseg_gsd_ud', 'ja')\
pipeline = Pipeline(stages=[
model = pipeline.fit(spark.createDataFrame([[""]]).toDF("text"))
example = spark.createDataFrame(pd.DataFrame({'text': ["""清代は湖北省が置かれ、そのまま現代の行政区分になっている。"""]}))
result = model.transform(example)
val word_segmenter = WordSegmenterModel.pretrained("wordseg_gsd_ud", "ja")
val pipeline = new Pipeline().setStages(Array(document_assembler, word_segmenter))
val result = pipeline.fit(Seq.empty["清代は湖北省が置かれそのまま現代の行政区分になっている。"].toDS.toDF("text")).transform(data)
import nlu

text = ["""清代は湖北省が置かれ、そのまま現代の行政区分になっている。"""]
token_df = nlu.load('ja.segment_words').predict(text, output_level='token')


|text                                                      |result                                                                                          |
|清代は湖北省が置かれ、そのまま現代の行政区分になっている。|[清代, は, 湖北, 省, が, 置か, れ, 、, その, まま, 現代, の, 行政, 区分, に, なっ, て, いる, 。]|

Model Information

Model Name: wordseg_gsd_ud
Compatibility: Spark NLP 2.7.0+
Edition: Official
Input Labels: [document]
Output Labels: [token]
Language: ja

Data Source

We trained this model on the the Universal Dependenicies data set from Google (GSD-UD).

Asahara, M., Kanayama, H., Tanaka, T., Miyao, Y., Uematsu, S., Mori, S., Matsumoto, Y., Omura, M., & Murawaki, Y. (2018). Universal Dependencies Version 2 for Japanese. In LREC-2018.


| Model         | precision    | recall       | f1-score     |
| JA_UD-GSD     |      0,7687  |      0,8048  |      0,7863  |