Spark NLP - Documentation and Reference

Concepts

Spark NLP Imports: Getting ready to work

Python
Scala

Since version 1.5.0 we are making necessary imports easy to reach, base will include general Spark NLP transformers and concepts, while annotator will include all annotators that we currently provide. We also need SparkML pipelines.
Example:

from sparknlp.base import *
from sparknlp.annotator import *

Since version 1.5.0 we are making necessary imports easy to reach, base._ will include general Spark NLP transformers and concepts, while annotator._ will include all annotators that we currently provide. We also need SparkML pipelines.
Example:

import com.johnsnowlabs.nlp.base._
import com.johnsnowlabs.nlp.annotator._

SparkML Pipelines are a uniform structure that helps creating and tuning practical machine learning pipelines. Spark NLP integrates with them seamlessly so it is important to have this concept handy. Once a Pipeline is trained with fit(), this becomes a PipelineModel
Example:

from pyspark.ml import Pipeline
pipeline = Pipeline().setStages([...])

import org.apache.spark.ml.Pipeline
new Pipeline().setStages(Array(...))

LightPipeline: A super-fast Spark-NLP pipeline for small data

Python
Scala

LightPipelines are Spark ML pipelines converted into a single machine but multithreaded task, becoming more than 10x times faster for smaller amounts of data (50k lines of text or below). To use them, simply plug in a trained (fitted) pipeline.
Example:

from sparknlp.base import LightPipeline
LightPipeline(someTrainedPipeline).annotate(someStringOrArray)

Functions:

annotate(string or string[]): returns dictionary list of annotation results
fullAnnotate(string or string[]): returns dictionary list of entire annotations content

import com.johnsnowlabs.nlp.LightPipeline
new LightPipeline(somePipelineModel).annotate(someStringOrArray))

Functions:

annotate(string or string[]): returns dictionary list of annotation results
fullAnnotate(string or string[]): returns dictionary list of entire annotations content

RecursivePipeline: A smarter Spark-NLP pipeline

Python
Scala

Recursive pipelines are SparkNLP specific pipelines that allow a Spark ML Pipeline to know about itself on every Pipeline Stage task, allowing annotators to utilize this same pipeline against external resources to process them in the same way the user decides. Only some of our annotators take advantage of this. RecursivePipeline behaves exactly the same than normal Spark ML pipelines, so they can be used with the same intention.
Example:

from sparknlp.annotator import *
recursivePipeline = RecursivePipeline(stages=[
        documentAssembler,
        sentenceDetector,
        tokenizer,
        lemmatizer,
        finisher
      ])

Recursive pipelines are SparkNLP specific pipelines that allow a Spark ML Pipeline to know about itself on every Pipeline Stage task, allowing annotators to utilize to use this same pipeline against external resources to process them in the same way the user decides. Only some of our annotators take advantage of this. RecursivePipeline behaves exactly the same than normal Spark ML pipelines, so they can be used with the same intention.
Example:

import com.johnsnowlabs.nlp.RecursivePipeline
val recursivePipeline = new RecursivePipeline()
      .setStages(Array(
        documentAssembler,
        sentenceDetector,
        tokenizer,
        lemmatizer,
        finisher
      ))

ExternalResource: Data properties outside the pipeline

Python
Scala

ExternalResource represent the properties of external data to be read, usually by the ResourceHelper (which is explained below). It contains information into how such external source may be read, and allows different protocols (hdfs, s3, etc) and formats (csv, text, parquet, etc). User does not usually need to create explicitly External Resources, but function parameters usually ask for elements used by it.
Example:

regex_matcher = RegexMatcher() \
    .setStrategy("MATCH_ALL") \
    .setExternalRules(path="/some/path", delimiter=",", read_as=ReadAs.LINE_BY_LINE, options={"format": "parquet"}) \
    .setOutputCol("regex")

Arguments:

path -> Takes a path with protocol of desintation file or folder
ReadAs -> "LINE_BY_LINE" or "SPARK_DATASET" will tell SparkNLP to use spark or not for this file or folder
options -> Contains information passed to Spark reader (e.g. format: "text") and other useful options for annotators (e.g. delimiter)

posTagger = new PerceptronApproach()
    .setInputCols(Array("sentence", "token"))
    .setOutputCol("pos")
    .setNIterations(1)
    .setCorpus(ExternalResource("/some/path/110CYL067.txt", ReadAs.LINE_BY_LINE, Map("delimiter" -> "|"))))

Arguments:

path -> Takes a path with protocol of desintation file or folder
ReadAs -> "LINE_BY_LINE" or "SPARK_DATASET" will tell SparkNLP to use spark or not for this file or folder
options -> Contains information passed to Spark reader (e.g. format: "text") and other useful options for annotators (e.g. delimiter)

ResourceHelper: Deal with data outside the pipeline

Python
Scala

When working with external resources, like training data that is not part of the pipeline process, our annotators use the ResourceHelper to efficiently parse and extract data into specific formats. This class may be utilized for other purposes by the user (Only in Scala)