Annotators

BertSentenceEmbeddings

BERT (Bidirectional Encoder Representations from Transformers) provides dense vector representations for natural language by using a deep, pre-trained neural network with the Transformer architecture

You can find the pre-trained models for BertEmbeddings in the Spark NLP Models repository

Output type: Sentence_Embeddings

Input types: Document

Reference: BertSentenceEmbeddings

Refer to the BertSentenceEmbeddings Scala docs for more

How to use pretrained BertEmbeddings:

bert = BertSentencembeddings.pretrained() \
      .setInputCols("document") \
      .setOutputCol("bert_sentence_embeddings")

val bert = BertEmbeddings.pretrained()
      .setInputCols("document")
      .setOutputCol("bert_sentence_embeddings")

YakeModel (Keywords Extraction)

Yake is an Unsupervised, Corpus-Independent, Domain and Language-Independent and Single-Document keyword extraction algorithm.

sExtracting keywords from texts has become a challenge for individuals and organizations as the information grows in complexity and size. The need to automate this task so that text can be processed in a timely and adequate manner has led to the emergence of automatic keyword extraction tools. Yake is a novel feature-based system for multi-lingual keyword extraction, which supports texts of different sizes, domain or languages. Unlike other approaches, Yake does not rely on dictionaries nor thesauri, neither is trained against any corpora. Instead, it follows an unsupervised approach which builds upon features extracted from the text, making it thus applicable to documents written in different languages without the need for further knowledge. This can be beneficial for a large number of tasks and a plethora of situations where access to training corpora is either limited or restricted.

The algorithm makes use of the position of a sentence and token. Therefore, to use the annotator, the text should be first sent through a Sentence Boundary Detector and then a tokenizer.

You can tweak the following parameters to get the best result from the annotator.

Output type: KEYWORD

Input types: TOKEN

Functions:

• setMinNGrams(int) Select the minimum length of a extracted keyword
• setMaxNGrams(int) Select the maximum length of a extracted keyword
• setNKeywords(int) Extract the top N keywords
• setStopWords(list) Set the list of stop words
• setThreshold(float) Each keyword will be given a keyword score greater than 0. (Lower the score better the keyword) Set an upper bound for the keyword score from this method.
• setWindowSize(int) Yake will construct a co-occurence matrix. You can set the window size for the cooccurence matrix construction from this method. ex: windowSize=2 will look at two words to both left and right of a candidate word.

Refer to the YakeModel Scala docs for more

keywords = YakeModel() \
    .setInputCols("token") \
    .setOutputCol("keywords") \
    .setMinNGrams(1) \
    .setMaxNGrams(3)\
    .setNKeywords(20)\
    .setStopWords(stopwords)

 val keywords = new YakeModel()
    .setInputCols("token")
    .setOutputCol("keywords")
    .setMinNGrams(1)
    .setMaxNGrams(3)
    .setNKeywords(20)
    .setStopWords(stopwords)

NER CRF (Named Entity Recognition CRF annotator)

This Named Entity recognition annotator allows for a generic model to be trained by utilizing a CRF machine learning algorithm. Its train data (train_ner) is either a labeled or an external CoNLL 2003 IOB based spark dataset with Annotations columns. Also the user has to provide word embeddings annotation column.
Optionally the user can provide an entity dictionary file for better accuracy
Output type: Named_Entity
Input types: Document, Token, POS, Word_Embeddings
Reference: NerCrfApproach | NerCrfModel
Functions:

• setLabelColumn: If DatasetPath is not provided, this Seq[Annotation] type of column should have labeled data per token
• setMinEpochs: Minimum number of epochs to train
• setMaxEpochs: Maximum number of epochs to train
• setL2: L2 regularization coefficient for CRF
• setC0: c0 defines decay speed for gradient
• setLossEps: If epoch relative improvement lass than this value, training is stopped
• setMinW: Features with less weights than this value will be filtered out
• setExternalFeatures(path, delimiter, readAs, options): Path to file or folder of line separated file that has something like this: Volvo:ORG with such delimiter, readAs LINE_BY_LINE or SPARK_DATASET with options passed to the latter.
• setEntities: Array of entities to recognize
• setVerbose: Verbosity level
• setRandomSeed: Random seed

Example:

Refer to the NerCrfApproach Scala docs for more details on the API.

nerTagger = NerCrfApproach()\
    .setInputCols(["sentence", "token", "pos", "embeddings"])\
    .setLabelColumn("label")\
    .setOutputCol("ner")\
    .setMinEpochs(1)\
    .setMaxEpochs(20)\
    .setLossEps(1e-3)\
    .setDicts(["ner-corpus/dict.txt"])\
    .setL2(1)\
    .setC0(1250000)\
    .setRandomSeed(0)\
    .setVerbose(2)
    .fit(train_ner)

val nerTagger = new NerCrfApproach()
    .setInputCols("sentence", "token", "pos", "embeddings")
    .setLabelColumn("label")
    .setMinEpochs(1)
    .setMaxEpochs(3)
    .setC0(34)
    .setL2(3.0)
    .setOutputCol("ner")
    .fit(trainNer)

NER DL (Named Entity Recognition Deep Learning annotator)

This Named Entity recognition annotator allows to train generic NER model based on Neural Networks. Its train data (train_ner) is either a labeled or an external CoNLL 2003 IOB based spark dataset with Annotations columns. Also the user has to provide word embeddings annotation column.
Neural Network architecture is Char CNNs - BiLSTM - CRF that achieves state-of-the-art in most datasets.
Output type: Named_Entity
Input types: Document, Token, Word_Embeddings
Reference: NerDLApproach | NerDLModel
Functions:

• setLabelColumn: If DatasetPath is not provided, this Seq[Annotation] type of column should have labeled data per token.
• setMaxEpochs: Maximum number of epochs to train.
• setLr: Initial learning rate.
• setPo: Learning rate decay coefficient. Real Learning Rate: lr / (1 + po * epoch).
• setBatchSize: Batch size for training.
• setDropout: Dropout coefficient.
• setVerbose: Verbosity level.
• setRandomSeed: Random seed.
• setOutputLogsPath: Folder path to save training logs.

Note: Please check here in case you get an IllegalArgumentException error with a description such as: Graph [parameter] should be [value]: Could not find a suitable tensorflow graph for embeddings dim: [value] tags: [value] nChars: [value]. Generate graph by python code in python/tensorflow/ner/create_models before usage and use setGraphFolder Param to point to output.

Example:

Refer to the NerDLApproach Scala docs for more details on the API.

nerTagger = NerDLApproach()\
    .setInputCols(["sentence", "token", "embeddings"])\
    .setLabelColumn("label")\
    .setOutputCol("ner")\
    .setMaxEpochs(10)\
    .setRandomSeed(0)\
    .setVerbose(2)
    .fit(train_ner)

val nerTagger = new NerDLApproach()
        .setInputCols("sentence", "token", "embeddings")
        .setOutputCol("ner")
        .setLabelColumn("label")
        .setMaxEpochs(120)
        .setRandomSeed(0)
        .setPo(0.03f)
        .setLr(0.2f)
        .setDropout(0.5f)
        .setBatchSize(9)
        .setVerbose(Verbose.Epochs)
        .fit(trainNer)

NER Converter (Converts IOB or IOB2 representation of NER to user-friendly)

NER Converter used to finalize work of NER annotators. Combines entites with types B-, I- and etc. to the Chunks with Named entity in the metadata field (if LightPipeline is used can be extracted after fullAnnotate())

This NER converter can be used to the output of a NER model into the ner chunk format.

Output type: Chunk Input types: Document, Token, Named_Entity Reference: NerConverter Functions:

• setWhiteList(Array(String)): If defined, list of entities to process. The rest will be ignored. Do not include IOB prefix on labels.
• setPreservePosition(Boolean): Whether to preserve the original position of the tokens in the original document or use the modified tokens.

Example:

Refer to the NerConverter Scala docs for more details on the API.

nerConverter = NerConverter()\
    .setInputCols(["sentence", "token", "ner_src"])\
    .setOutputCol("ner_chunk")

val nerConverter = new NerConverter()
        .setInputCols("sentence", "token", "ner_src")
        .setOutputCol("ner_chunk")

Norvig SpellChecker

This annotator retrieves tokens and makes corrections automatically if not found in an English dictionary
Output type: Token Input types: Token Inputs: Any text for corpus. A list of words for dictionary. A comma separated custom dictionary. Train Data: train_corpus is a spark dataset of text content Reference: NorvigSweetingApproach | NorvigSweetingModel
Functions:

• setDictionary(path, tokenPattern, readAs, options): path to file with properly spelled words, tokenPattern is the regex pattern to identify them in text, readAs LINE_BY_LINE or SPARK_DATASET, with options passed to Spark reader if the latter is set.
• setCaseSensitive(boolean): defaults to false. Might affect accuracy
• setDoubleVariants(boolean): enables extra check for word combinations, more accuracy at performance
• setShortCircuit(boolean): faster but less accurate mode
• setWordSizeIgnore(int): Minimum size of word before moving on. Defaults to 3.
• setDupsLimit(int): Maximum duplicate of characters to account for. Defaults to 2.
• setReductLimit(int): Word reduction limit. Defaults to 3
• setIntersections(int): Hamming intersections to attempt. Defaults to 10.
• setVowelSwapLimit(int): Vowel swap attempts. Defaults to 6.

Example:

Refer to the NorvigSweetingApproach Scala docs for more details on the API.

spell_checker = NorvigSweetingApproach() \
    .setInputCols(["token"]) \
    .setOutputCol("checked") \
    .setDictionary("coca2017.txt", "[a-zA-Z]+")

val symSpellChecker = new NorvigSweetingApproach()
      .setInputCols("token")
      .setOutputCol("checked")
      .setDictionary("coca2017.txt", "[a-zA-Z]+")

Symmetric SpellChecker

This spell checker is inspired on Symmetric Delete algorithm. It retrieves tokens and utilizes distance metrics to compute possible derived words
Output type: Token
Input types: Token
Inputs: Any text for corpus. A list of words for dictionary. A comma separated custom dictionary.
Train Data: train_corpus is a spark dataset of text content
Reference: SymmetricDeleteApproach | SymmetricDeleteModel
Functions:

• setDictionary(path, tokenPattern, readAs, options): Optional dictionary of properly written words. If provided, significantly boosts spell checking performance
• setMaxEditDistance(distance): Maximum edit distance to calculate possible derived words. Defaults to 3.

Example:

Refer to the SymmetricDeleteApproach Scala docs for more details on the API.

spell_checker = SymmetricDeleteApproach() \
    .setInputCols(["token"]) \
    .setOutputCol("spell") \
    .fit(train_corpus)

val spellChecker = new SymmetricDeleteApproach()
    .setInputCols(Array("normalized"))
    .setOutputCol("spell")
    .fit(trainCorpus)

Context SpellChecker

Implements Noisy Channel Model Spell Algorithm. Correction candidates are extracted combining context information and word information
Output type: Token
Input types: Token
Inputs: Any text for corpus. A list of words for dictionary. A comma separated custom dictionary.
Train Data: train_corpus is a spark dataset of text content
Reference: ContextSpellCheckerApproach | ContextSpellCheckerModel
Functions:

• setLanguageModelClasses(languageModelClasses:Int): Number of classes to use during factorization of the softmax output in the LM. Defaults to 2000.
• setWordMaxDistance(dist:Int): Maximum distance for the generated candidates for every word. Defaults to 3.
• setMaxCandidates(candidates:Int): Maximum number of candidates for every word. Defaults to 6.
• setCaseStrategy(strategy:Int): What case combinations to try when generating candidates. ALL_UPPER_CASE = 0, FIRST_LETTER_CAPITALIZED = 1, ALL = 2. Defaults to 2.
• setErrorThreshold(threshold:Float): Threshold perplexity for a word to be considered as an error. Defaults to 10f.
• setTradeoff(alpha:Float): Tradeoff between the cost of a word error and a transition in the language model. Defaults to 18.0f.
• setMaxWindowLen(length:Integer): Maximum size for the window used to remember history prior to every correction. Defaults to 5.
• setGamma(g:Float): Controls the influence of individual word frequency in the decision.
• updateVocabClass(label:String, vocab:Array(String), append:boolean): Update existing vocabulary classes so they can cover new words. If append set to false overwrite vocabulary class in the model by new words, if true extends existing vocabulary class. Defaults to true.
• updateRegexClass(label:String, regex:String): Update existing regex rule for the class defined by regex.

Train:

• setWeightedDistPath(weightedDistPath:String): The path to the file containing the weights for the levenshtein distance.
• setEpochs(epochs:Int): Number of epochs to train the language model. Defaults to 2.
• setInitialBatchSize(batchSize:Int): Batch size for the training in NLM. Defaults to 24.
• setInitialRate(initialRate:Float): Initial learning rate for the LM. Defaults to .7f.
• setFinalRate(finalRate:Float): Final learning rate for the LM. Defaults to 0.0005f.
• setValidationFraction(validationFraction:Float): Percentage of datapoints to use for validation. Defaults to .1f.
• setMinCount(minCount:Float): Min number of times a token should appear to be included in vocab. Defaults to 3.0f.
• setCompoundCount(compoundCount:Int): Min number of times a compound word should appear to be included in vocab. Defaults to 5.
• setClassCount(classCount:Int): Min number of times the word need to appear in corpus to not be considered of a special class. Defaults to 15.

Example:

Refer to the ContextSpellCheckerApproach Scala docs for more details on the API.

spell_checker = ContextSpellCheckerApproach() \
    .setInputCols(["token"]) \
    .setOutputCol("spell") \
    .fit(train_corpus)

val spellChecker = new ContextSpellCheckerApproach()
    .setInputCols(Array("token"))
    .setOutputCol("spell")
    .fit(trainCorpus)

Dependency Parsers

Dependency parser provides information about word relationship. For example, dependency parsing can tell you what the subjects and objects of a verb are, as well as which words are modifying (describing) the subject. This can help you find precise answers to specific questions. The following diagram illustrates a dependency-style analysis using the standard graphical method favored in the dependency-parsing community.

Relations among the words are illustrated above the sentence with directed, labeled arcs from heads to dependents. We call this a typed dependency structure because the labels are drawn from a fixed inventory of grammatical relations. It also includes a root node that explicitly marks the root of the tree, the head of the entire structure. [1]

Untyped Dependency Parser (Unlabeled grammatical relation)

Unlabeled parser that finds a grammatical relation between two words in a sentence. Its input is a directory with dependency treebank files.
Output type: Dependency
Input types: Document, POS, Token
Reference: DependencyParserApproach | DependencyParserModel
Functions:

• setNumberOfIterations: Number of iterations in training, converges to better accuracy
• setDependencyTreeBank: Dependency treebank folder with files in Penn Treebank format
• conllU: Path to a file in CoNLL-U format

Example:

Refer to the DependencyParserApproach Scala docs for more details on the API.

dependency_parser = DependencyParserApproach() \
            .setInputCols(["sentence", "pos", "token"]) \
            .setOutputCol("dependency") \
            .setDependencyTreeBank("file://parser/dependency_treebank") \
            .setNumberOfIterations(10)

val dependencyParser = new DependencyParserApproach()
    .setInputCols(Array("sentence", "pos", "token"))
    .setOutputCol("dependency")
    .setDependencyTreeBank("parser/dependency_treebank")
    .setNumberOfIterations(10)

Typed Dependency Parser (Labeled grammatical relation)

Labeled parser that finds a grammatical relation between two words in a sentence. Its input is a CoNLL2009 or ConllU dataset.
Output type: Labeled Dependency
Input types: Token, POS, Dependency
Reference: TypedDependencyParserApproach | TypedDependencyParserModel
Functions:

• setNumberOfIterations: Number of iterations in training, converges to better accuracy
• setConll2009: Path to a file in CoNLL 2009 format
• setConllU: Path to a file in CoNLL-U format

Example:

Refer to the TypedDependencyParserApproach Scala docs for more details on the API.

typed_dependency_parser = TypedDependencyParserApproach() \
            .setInputCols(["token", "pos", "dependency"]) \
            .setOutputCol("labdep") \
            .setConll2009("file://conll2009/eng.train") \
            .setNumberOfIterations(10)

val typedDependencyParser = new TypedDependencyParserApproach()
    .setInputCols(Array("token", "pos", "dependency"))
    .setOutputCol("labdep")
    .setConll2009("conll2009/eng.train"))

References

[1] Speech and Language Processing. Daniel Jurafsky & James H. Martin. 2018