Japanese Lemmatizer

Description

This model uses context and language knowledge to assign all forms and inflections of a word to a single root. This enables the pipeline to treat the past and present tense of a verb, for example, as the same word instead of two completely different words. The lemmatizer takes into consideration the context surrounding a word to determine which root is correct when the word form alone is ambiguous.

Live Demo Open in Colab Download

How to use

document_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

word_segmenter = WordSegmenterModel.pretrained('wordseg_gsd_ud', 'ja')\
        .setInputCols("document")\
        .setOutputCol("token")

lemmatizer = LemmatizerModel.pretrained("lemma", "ja") \
        .setInputCols(["token"]) \
        .setOutputCol("lemma")

nlp_pipeline = Pipeline(stages=[document_assembler, word_segmenter , lemmatizer])
light_pipeline = LightPipeline(nlp_pipeline.fit(spark.createDataFrame([[""]]).toDF("text")))
results = light_pipeline.fullAnnotate(["これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。"])

val document_assembler = DocumentAssembler()
    .setInputCol("text")
    .setOutputCol("document")

val word_segmenter = WordSegmenterModel.pretrained('wordseg_gsd_ud', 'ja')
        .setInputCols("document")
        .setOutputCol("token")

val lemmatizer = LemmatizerModel.pretrained("lemma", "ja")
        .setInputCols(["token"])
        .setOutputCol("lemma")

val pipeline = new Pipeline().setStages(Array(document_assembler, word_segmenter , lemmatizer))
val result = pipeline.fit(Seq.empty["これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。"].toDS.toDF("text")).transform(data)
import nlu

text = ["これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。"]
lemma_df = nlu.load('ja.lemma').predict(text, output_level = "document")
lemma_df.lemma.values[0]

Results

{'lemma': [Annotation(token, 0, 1, これ, {'sentence': '0'}),
   Annotation(token, 2, 2, にる, {'sentence': '0'}),
   Annotation(token, 3, 4, 不快, {'sentence': '0'}),
   Annotation(token, 5, 5, 感, {'sentence': '0'}),
   Annotation(token, 6, 6, を, {'sentence': '0'}),
   Annotation(token, 7, 8, 示す, {'sentence': '0'}),
   Annotation(token, 9, 10, 住民, {'sentence': '0'}),
   Annotation(token, 11, 11, はる, {'sentence': '0'}),
   Annotation(token, 12, 12, いる, {'sentence': '0'}),
   Annotation(token, 13, 14, まする, {'sentence': '0'}),
   Annotation(token, 15, 15, たる, {'sentence': '0'}),
   Annotation(token, 16, 16, がる, {'sentence': '0'}),
   Annotation(token, 17, 17, ,, {'sentence': '0'}),
   Annotation(token, 18, 19, 現在, {'sentence': '0'}),
   Annotation(token, 20, 20, ,, {'sentence': '0'}),
   Annotation(token, 21, 23, 表立つ, {'sentence': '0'}),
   Annotation(token, 24, 24, てる, {'sentence': '0'}),
   Annotation(token, 25, 26, 反対, {'sentence': '0'}),
   Annotation(token, 27, 27, やる, {'sentence': '0'}),
   Annotation(token, 28, 29, 抗議, {'sentence': '0'}),
   Annotation(token, 30, 30, のる, {'sentence': '0'}),
   Annotation(token, 31, 31, 声, {'sentence': '0'}),
   Annotation(token, 32, 32, を, {'sentence': '0'}),
   Annotation(token, 33, 34, 挙げる, {'sentence': '0'}),
   Annotation(token, 35, 35, てる, {'sentence': '0'}),
   Annotation(token, 36, 37, いる, {'sentence': '0'}),
   Annotation(token, 38, 39, 住民, {'sentence': '0'}),
   Annotation(token, 40, 40, はる, {'sentence': '0'}),
   Annotation(token, 41, 41, いる, {'sentence': '0'}),
   Annotation(token, 42, 43, なぐ, {'sentence': '0'}),
   Annotation(token, 44, 45, よう, {'sentence': '0'}),
   Annotation(token, 46, 47, です, {'sentence': '0'}),
   Annotation(token, 48, 48, 。, {'sentence': '0'})]}

Model Information

Model Name: lemma
Compatibility: Spark NLP 2.7.0+
Edition: Official
Input Labels: [document]
Output Labels: [token]
Language: ja

Data Source

The model was trained using the universal dependencies data set version 2 and the IPADIC dictionary from Mecab.

References:

  • Asahara, M., Kanayama, H., Tanaka, T., Miyao, Y., Uematsu, S., Mori, S., Matsumoto, Y., Omura, M., & Murawaki, Y. (2018). Universal Dependencies Version 2 for Japanese. In LREC-2018.