Word Segmenter for Chinese

Description

WordSegmenterModel-WSM is based on maximum entropy probability model to detect word boundaries in Chinese text. Chinese text is written without white space between the words, and a computer-based application cannot know a priori which sequence of ideograms form a word. In many natural language processing tasks such as part-of-speech (POS) and named entity recognition (NER) require word segmentation as a initial step.

Open in Colab Download Copy S3 URI

How to use


word_segmenter = WordSegmenterModel.pretrained("wordseg_ctb9", "zh")        .setInputCols(["sentence"])        .setOutputCol("token")
pipeline = Pipeline(stages=[document_assembler, word_segmenter])
ws_model = pipeline.fit(spark.createDataFrame([[""]]).toDF("text"))
example = spark.createDataFrame([['从John Snow Labs你好! ']], ["text"])
result = ws_model.transform(example)


val word_segmenter = WordSegmenterModel.pretrained("wordseg_ctb9", "zh")
        .setInputCols(Array("sentence"))
        .setOutputCol("token")
val pipeline = new Pipeline().setStages(Array(document_assembler, word_segmenter))
val data = Seq("从John Snow Labs你好! ").toDF("text")
val result = pipeline.fit(data).transform(data)


import nlu
text = [""从John Snow Labs你好 ""]
token_df = nlu.load('zh.segment_words.ctb9').predict(text)
token_df
    

Results


0        从
1        J
2        o
3        h
4        n
5        S
6        n
7        o
8        w
9     Labs
10       你
11       好
12       !
Name: token, dtype: object

Model Information

Model Name: wordseg_ctb9
Compatibility: Spark NLP 3.0.0+
License: Open Source
Edition: Official
Input Labels: [document]
Output Labels: [words_segmented]
Language: zh