Part of Speech for Korean

Description

This model annotates the part of speech of tokens in a text. The parts of speech annotated include PRON (pronoun), CCONJ (coordinating conjunction), and 13 others. The part of speech model is useful for extracting the grammatical structure of a piece of text automatically.

Predicted Entities

ADJ, ADP, ADV, AUX, CONJ, DET, NOUN, NUM, PART, PRON, PROPN, PUNCT, SYM, VERB, and X.

Live Demo Open in Colab Download

How to use

Use as part of an nlp pipeline after tokenization.

document_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")
    
sentence_detector = SentenceDetector()\
    .setInputCols(["document"])\
    .setOutputCol("sentence")
    
word_segmenter = WordSegmenterModel.pretrained("wordseg_kaist_ud", "ko")\
        .setInputCols(["sentence"])\
        .setOutputCol("token")
        
pos = PerceptronModel.pretrained("pos_ud_kaist", "ko") \
    .setInputCols(["document", "token"]) \
    .setOutputCol("pos")

pipeline = Pipeline(stages=[
        document_assembler,
        sentence_detector,
        word_segmenter,
        posTagger
    ])

example = spark.createDataFrame(pd.DataFrame({'text': ["비파를탄주하는그늙은명인의시는아름다운화음이었고완벽한음악으로순간적인조화를이룬세계의울림이었다."]}))

result = pipeline.fit(example).transform(example)
val document_assembler = DocumentAssembler()
        .setInputCol("text")
        .setOutputCol("document")
        
val sentence_detector = SentenceDetector()
        .setInputCols(["document"])
        .setOutputCol("sentence")
        
val word_segmenter = WordSegmenterModel.pretrained("wordseg_kaist_ud", "ko")
        .setInputCols(["sentence"])
        .setOutputCol("token")
        
val pos = PerceptronModel.pretrained("pos_ud_kaist", "ko")
    .setInputCols(Array("document", "token"))
    .setOutputCol("pos")

val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, word_segmenter, pos))

val result = pipeline.fit(Seq.empty["비파를탄주하는그늙은명인의시는아름다운화음이었고완벽한음악으로순간적인조화를이룬세계의울림이었다."].toDS.toDF("text")).transform(data)
import nlu

text = ["""비파를탄주하는그늙은명인의시는아름다운화음이었고완벽한음악으로순간적인조화를이룬세계의울림이었다."""]
pos_df = nlu.load('ko.pos.ud_kaist').predict(text, output_level='token')
pos_df

Results

+----------+-----+
|token     |pos  |
+----------+-----+
|비파를    |NOUN |
|탄주하는  |VERB |
|그        |DET  |
|늙은      |VERB |
|명인의    |NOUN |
|시는      |NOUN |
|아름다운  |ADJ  |
|화음이었고|CCONJ|
|완벽한    |VERB |
|음악으로  |NOUN |
|순간적인  |VERB |
|조화를    |NOUN |
|이룬      |VERB |
|세계의    |NOUN |
|울림이었다|VERB |
|.         |PUNCT|
+----------+-----+

Model Information

Model Name: pos_ud_kaist
Compatibility: Spark NLP 2.7.0+
Edition: Official
Input Labels: [sentence, token]
Output Labels: [pos]
Language: ko

Data Source

The model was trained in the Universal Dependencies, curated by the Korea Advanced Institute of Science and Technology (KAIST)

Reference:

> Building Universal Dependency Treebanks in Korean, Jayeol Chun, Na-Rae Han, Jena D. Hwang, and Jinho D. Choi. 
In Proceedings of the 11th International Conference on Language Resources and Evaluation, LREC'18, Miyazaki, Japan, 2018.

Benchmarking

|              | precision | recall | f1-score | support |
|--------------|-----------|--------|----------|---------|
| ADJ          | 0.90      | 0.81   | 0.85     | 1180    |
| ADP          | 0.95      | 0.97   | 0.96     | 160     |
| ADV          | 0.89      | 0.82   | 0.85     | 4156    |
| AUX          | 0.85      | 0.84   | 0.84     | 1074    |
| CCONJ        | 0.82      | 0.71   | 0.76     | 1471    |
| DET          | 0.92      | 0.91   | 0.91     | 272     |
| INTJ         | 0.00      | 0.00   | 0.00     | 3       |
| NOUN         | 0.80      | 0.93   | 0.86     | 8338    |
| NUM          | 0.86      | 0.91   | 0.88     | 631     |
| PART         | 0.00      | 0.00   | 0.00     | 18      |
| PRON         | 0.96      | 0.91   | 0.93     | 405     |
| PROPN        | 0.85      | 0.58   | 0.69     | 1377    |
| PUNCT        | 1.00      | 1.00   | 1.00     | 3109    |
| SCONJ        | 0.84      | 0.72   | 0.78     | 1547    |
| SYM          | 1.00      | 0.98   | 0.99     | 115     |
| VERB         | 0.87      | 0.87   | 0.87     | 4378    |
| X            | 0.94      | 0.71   | 0.81     | 132     |
| accuracy     | 0.86      | 28366  |          |         |
| macro avg    | 0.79      | 0.74   | 0.76     | 28366   |
| weighted avg | 0.86      | 0.86   | 0.86     | 28366   |