Annotators

NER CRF (Named Entity Recognition CRF annotator)

This Named Entity recognition annotator allows for a generic model to be trained by utilizing a CRF machine learning algorithm. Its train data (train_ner) is either a labeled or an external CoNLL 2003 IOB based spark dataset with Annotations columns. Also the user has to provide word embeddings annotation column.
Optionally the user can provide an entity dictionary file for better accuracy

Output Annotator Type: Named_Entity

Input Annotator Types: Document, Token, POS, Word_Embeddings

Example:

nerTagger = NerCrfApproach()\
        .setInputCols(["sentence", "token", "pos", "embeddings"])\
        .setLabelColumn("label")\
        .setOutputCol("ner")\
        .setMinEpochs(1)\
        .setMaxEpochs(20)\
        .setLossEps(1e-3)\
        .setDicts(["ner-corpus/dict.txt"])\
        .setL2(1)\
        .setC0(1250000)\
        .setRandomSeed(0)\
        .setVerbose(2)

val nerTagger = new NerCrfApproach()
        .setInputCols("sentence", "token", "pos", "embeddings")
        .setLabelColumn("label")
        .setMinEpochs(1)
        .setMaxEpochs(3)
        .setC0(34)
        .setL2(3.0)
        .setOutputCol("ner")

NER DL (Named Entity Recognition Deep Learning annotator)

This Named Entity recognition annotator allows to train generic NER model based on Neural Networks. Its train data (train_ner) is either a labeled or an external CoNLL 2003 IOB based spark dataset with Annotations columns. Also the user has to provide word embeddings annotation column.
Neural Network architecture is Char CNNs - BiLSTM - CRF that achieves state-of-the-art in most datasets.

Output Annotator Type: Named_Entity

Input Annotator Types: Document, Token, Word_Embeddings

Note: Please check here in case you get an IllegalArgumentException error with a description such as:

Graph [parameter] should be [value]: Could not find a suitable tensorflow graph for embeddings dim: [value] tags: [value] nChars: [value]. Generate graph by python code in python/tensorflow/ner/create_models before usage and use setGraphFolder Param to point to output.

Example:

nerTagger = NerDLApproach()\
        .setInputCols(["sentence", "token", "embeddings"])\
        .setLabelColumn("label")\
        .setOutputCol("ner")\
        .setMaxEpochs(10)\
        .setRandomSeed(0)\
        .setVerbose(2)

val nerTagger = new NerDLApproach()
        .setInputCols("sentence", "token", "embeddings")
        .setOutputCol("ner")
        .setLabelColumn("label")
        .setMaxEpochs(120)
        .setRandomSeed(0)
        .setPo(0.03f)
        .setLr(0.2f)
        .setDropout(0.5f)
        .setBatchSize(9)
        .setVerbose(Verbose.Epochs)

NER Converter (Converts IOB or IOB2 representation of NER to user-friendly)

NER Converter used to finalize work of NER annotators. Combines entites with types B-, I- and etc. to the Chunks with Named entity in the metadata field (if LightPipeline is used can be extracted after fullAnnotate())

This NER converter can be used to the output of a NER model into the ner chunk format.

Output Annotator Type: Chunk

Input Annotator Types: Document, Token, Named_Entity

Example:

nerConverter = NerConverter()\
        .setInputCols(["sentence", "token", "ner_src"])\
        .setOutputCol("ner_chunk")

val nerConverter = new NerConverter()
        .setInputCols("sentence", "token", "ner_src")
        .setOutputCol("ner_chunk")

Norvig SpellChecker

This annotator retrieves tokens and makes corrections automatically if not found in an English dictionary

Output Annotator Type: Token

Input Annotator Types: Token

Inputs: Any text for corpus. A list of words for dictionary. A comma separated custom dictionary.

Train Data: train_corpus is a spark dataset of text content

Example:

spell_checker = NorvigSweetingApproach() \
        .setInputCols(["token"]) \
        .setOutputCol("checked") \
        .setDictionary("coca2017.txt", "[a-zA-Z]+")

val symSpellChecker = new NorvigSweetingApproach()
        .setInputCols("token")
        .setOutputCol("checked")
        .setDictionary("coca2017.txt", "[a-zA-Z]+")

Symmetric SpellChecker

This spell checker is inspired on Symmetric Delete algorithm. It retrieves tokens and utilizes distance metrics to compute possible derived words

Output Annotator Type: Token

Input Annotator Types: Token

Inputs: Any text for corpus. A list of words for dictionary. A comma separated custom dictionary.

Train Data: train_corpus is a spark dataset of text content

Example:

spell_checker = SymmetricDeleteApproach() \
    .setInputCols(["token"]) \
    .setOutputCol("spell")

val spellChecker = new SymmetricDeleteApproach()
    .setInputCols(Array("normalized"))
    .setOutputCol("spell")

Context SpellChecker

Implements Noisy Channel Model Spell Algorithm. Correction candidates are extracted combining context information and word information

Output Annotator Type: Token

Input Annotator Types: Token

Inputs: Any text for corpus. A list of words for dictionary. A comma separated custom dictionary.

Train Data: train_corpus is a spark dataset of text content

Example:

spell_checker = ContextSpellCheckerApproach() \
    .setInputCols(["token"]) \
    .setOutputCol("spell") \
    .fit(train_corpus) \
    .setErrorThreshold(4.0)\
    .setTradeoff(6.0)

val spellChecker = new ContextSpellCheckerApproach()
    .setInputCols(Array("token"))
    .setOutputCol("spell")
    .fit(trainCorpus)
    .setErrorThreshold(4.0)
    .setTradeoff(6.0)

Dependency Parsers

Dependency parser provides information about word relationship. For example, dependency parsing can tell you what the subjects and objects of a verb are, as well as which words are modifying (describing) the subject. This can help you find precise answers to specific questions. The following diagram illustrates a dependency-style analysis using the standard graphical method favored in the dependency-parsing community.

Relations among the words are illustrated above the sentence with directed, labeled arcs from heads to dependents. We call this a typed dependency structure because the labels are drawn from a fixed inventory of grammatical relations. It also includes a root node that explicitly marks the root of the tree, the head of the entire structure. [1]

Untyped Dependency Parser (Unlabeled grammatical relation)

Unlabeled parser that finds a grammatical relation between two words in a sentence. Its input is a directory with dependency treebank files.

Output Annotator Type: Dependency

Input Annotator Types: Document, POS, Token

Example:

dependency_parser = DependencyParserApproach() \
        .setInputCols(["sentence", "pos", "token"]) \
        .setOutputCol("dependency") \
        .setDependencyTreeBank("file://parser/dependency_treebank") \
        .setNumberOfIterations(10)

val dependencyParser = new DependencyParserApproach()
        .setInputCols(Array("sentence", "pos", "token"))
        .setOutputCol("dependency")
        .setDependencyTreeBank("parser/dependency_treebank")
        .setNumberOfIterations(10)

Typed Dependency Parser (Labeled grammatical relation)

Labeled parser that finds a grammatical relation between two words in a sentence. Its input is a CoNLL2009 or ConllU dataset.

Output Annotator Type: Labeled Dependency

Input Annotator Types: Token, POS, Dependency

Example:

typed_dependency_parser = TypedDependencyParserApproach() \
        .setInputCols(["token", "pos", "dependency"]) \
        .setOutputCol("labdep") \
        .setConll2009("file://conll2009/eng.train") \
        .setNumberOfIterations(10)

val typedDependencyParser = new TypedDependencyParserApproach()
        .setInputCols(Array("token", "pos", "dependency"))
        .setOutputCol("labdep")
        .setConll2009("conll2009/eng.train"))
        

References

[1] Speech and Language Processing. Daniel Jurafsky & James H. Martin. 2018