Part of Speech for Japanese

Description

This model annotates the part of speech of tokens in a text. The parts of speech annotated include PRON (pronoun), CCONJ (coordinating conjunction), and 13 others. The part of speech model is useful for extracting the grammatical structure of a piece of text automatically.

Live Demo Open in Colab Download

How to use

document_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")
    
sentence_detector = SentenceDetector()\
    .setInputCols(["document"])\
    .setOutputCol("sentence")
    
word_segmenter = WordSegmenterModel.pretrained("wordseg_gsd_ud", "ja")\
        .setInputCols(["sentence"])\
        .setOutputCol("token")
        
pos = PerceptronModel.pretrained("pos_ud_gsd", "ja") \
    .setInputCols(["document", "token"]) \
    .setOutputCol("pos")

pipeline = Pipeline(stages=[
        document_assembler,
        sentence_detector,
        word_segmenter,
        posTagger
    ])

example = spark.createDataFrame(pd.DataFrame({'text': ["""院長と話をしたところ、腰痛治療も得意なようです。"""]}))

result = pipeline.fit(example).transform(example)

val document_assembler = DocumentAssembler()
        .setInputCol("text")
        .setOutputCol("document")
        
val sentence_detector = SentenceDetector()
        .setInputCols(["document"])
        .setOutputCol("sentence")
        
val word_segmenter = WordSegmenterModel.pretrained("wordseg_gsd_ud", "ja")
        .setInputCols(["sentence"])
        .setOutputCol("token")
        
val pos = PerceptronModel.pretrained("pos_ud_gsd", "ja")
    .setInputCols(Array("document", "token"))
    .setOutputCol("pos")

val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, word_segmenter, pos))

val result = pipeline.fit(Seq.empty["院長と話をしたところ腰痛治療も得意なようです。"].toDS.toDF("text")).transform(data)
import nlu

text = ["""5月13日に放送されるフジテレビ系「僕らの音楽」にて、福原美穂とAIという豪華共演が決定した。"""]
pos_df = nlu.load('ja.pos.ud_gsd').predict(text, output_level='token')
pos_df

Results

+------+-----+
|token |pos  |
+------+-----+
|院長  |NOUN |
|と    |ADP  |
|話    |NOUN |
|を    |ADP  |
|し    |VERB |
|た    |AUX  |
|ところ|NOUN |
|、    |PUNCT|
|腰痛  |NOUN |
|治療  |NOUN |
|も    |ADP  |
|得意  |ADJ  |
|な    |AUX  |
|よう  |AUX  |
|です  |AUX  |
|。    |PUNCT|
+------+-----+

Model Information

Model Name: pos_ud_gsd
Compatibility: Spark NLP 2.7.0+
Edition: Official
Input Labels: [sentence, token]
Output Labels: [pos]
Language: ja

Data Source

The model was trained on the Universal Dependencies, curated by Google.

Reference:

> Asahara, M., Kanayama, H., Tanaka, T., Miyao, Y., Uematsu, S., Mori, S., Matsumoto, Y., Omura, M., & Murawaki, Y. (2018). 
Universal Dependencies Version 2 for Japanese. In LREC-2018.

Benchmarking

| pos_tag      | precision | recall | f1-score | support |
|--------------|-----------|--------|----------|---------|
| ADJ          | 0.90      | 0.78   | 0.84     | 350     |
| ADP          | 0.98      | 0.99   | 0.99     | 2804    |
| ADV          | 0.87      | 0.65   | 0.74     | 220     |
| AUX          | 0.95      | 0.98   | 0.96     | 1768    |
| CCONJ        | 0.97      | 0.93   | 0.95     | 42      |
| DET          | 1.00      | 1.00   | 1.00     | 66      |
| INTJ         | 0.00      | 0.00   | 0.00     | 1       |
| NOUN         | 0.93      | 0.98   | 0.95     | 3692    |
| NUM          | 0.99      | 0.98   | 0.99     | 251     |
| PART         | 0.96      | 0.83   | 0.89     | 128     |
| PRON         | 0.97      | 0.94   | 0.95     | 101     |
| PROPN        | 0.92      | 0.70   | 0.79     | 313     |
| PUNCT        | 1.00      | 1.00   | 1.00     | 1294    |
| SCONJ        | 0.97      | 0.94   | 0.96     | 682     |
| SYM          | 0.99      | 1.00   | 0.99     | 67      |
| VERB         | 0.96      | 0.92   | 0.94     | 1255    |
| accuracy     | 0.96      | 13034  |          |         |
| macro avg    | 0.90      | 0.85   | 0.87     | 13034   |
| weighted avg | 0.96      | 0.96   | 0.95     | 13034   |