SparkNLP - Quick Start

Last updated:


SparkNLP is built on top of Apache Spark 2.3.0 and works with any user provided Spark 2.x.x it is advised to have basic knowledge of the framework and a working environment before using Spark-NLP. Refer to its documentation to get started with Spark.

To start using the library, execute any of the following lines depending on your desired use case:

spark-shell --packages JohnSnowLabs:spark-nlp:1.5.4
pyspark --packages JohnSnowLabs:spark-nlp:1.5.4
spark-submit --packages JohnSnowLabs:spark-nlp:1.5.4

Using Databricks cloud cluster or an Apache Zeppelin Scala notebook? Add the following Maven coordinates in the appropriate menu:


For Python in Apache Zeppelin you may need to setup SPARK_SUBMIT_OPTIONS utilizing --packages instruction shown above like this

export SPARK_SUBMIT_OPTIONS="--packages JohnSnowLabs:spark-nlp:1.5.4"

Python Jupyter Notebook with PySpark? add the following environment variables (depending on your OS)

export SPARK_HOME=/path/to/your/spark/folder

pyspark --packages JohnSnowLabs:spark-nlp:1.5.4

Python without explicit Spark Installationk? Use pip to install (after you pip installed pyspark)

pip install --index-url spark-nlp==1.5.4

In this way, you will have to start SparkSession in your python program manually, this is an example

spark = SparkSession.builder \
    .config("spark.driver.maxResultSize", "2G") \
    .config("spark.driver.extraClassPath", "lib/spark-nlp-assembly-1.5.4.jar")\
    .config("spark.kryoserializer.buffer.max", "500m")\

Pre-compiled Spark-NLP assembly fat-jar for using in standalone projects, may be downloaded here Non-fat-jar may be downloaded here then, run spark-shell or spark-submit with appropriate --jars /path/to/spark-nlp_2.11-1.5.4.jar to use the library in spark.

For further alternatives and documentation check out our README page in GitHub.


Spark ML provides a set of Machine Learning applications, and it's logic consists of two main components: Estimators and Transformers. The first, have a method called fit() which secures and trains a piece of data to such application, and a Transformer, which is generally the result of a fitting process, applies changes to the the target dataset. These components have been embedded to be applicable to Spark NLP. Pipelines are a mechanism that allow multiple estimators and transformers within a single workflow, allowing multiple chained transformations along a Machine Learning task. Refer to SparkML library for more information.


An annotation is the basic form of the result of a Spark-NLP operation. It's structure is made of:

  • annotatorType: which annotator generated this annotation
  • begin: the begin of the matched content relative to raw-text
  • end: the end of the matched content relative to raw-text
  • metadata: content of matched result and additional information

This object is automatically generated by annotators after a transform process. No manual work is required. But it must be understood in order to use it efficiently.


Annotators are the spearhead of NLP functionalities in SparkNLP. There are two forms of annotators:

  • Annotator Approaches: Are those who represent a Spark ML Estimator and require a training stage. They have a function called fit(data) which trains a model based on some data. They produce the second type of annotator which is an annotator model or transformer.
  • Annotator Model: They are spark models or transformers, meaning they have a transform(data) function which take a dataset and add to it a column with the result of this annotation. All transformers are additive, meaning they append to current data, never replace or delete previous information.

Both forms of annotators can be included in a Pipeline and will automatically go through all stages in the provided order and transform the data accordingly. A Pipeline is turned into a PipelineModel after the fit() stage. Either before or after can be saved and re-loaded to disk at any time.

Common Functions

  • setInputCols(column_names): Takes a list of column names of annotations required by this annotator
  • setOutputCol(column_name): Defines the name of the column containing the result of this annotator. Use this name as an input for other annotators requiring the annotations of this one.

Quickly annotate some text

Basic Pipeline

A basic pipeline is a downloadable Spark ML pipeline ready to compute some annotations for you right away. Keep in mind this is trained with generic data and is not meant for production use, but should give you an insight of how the library works at a glance.

Downloading and using a pretrained pipeline

Basic pipelines allow to inject either common strings or Spark dataframes. Either way, SparkNLP will execute the right pipeline type to bring back the results. Let's retrieve it:

import com.johnsnowlabs.nlp.pretrained.pipelines.en.BasicPipeline
BasicPipeline().annotate(["We are very happy about SparkNLP", "And this is just another sentence")

Using a pretrained pipeline with spark dataframes

Since this is Apache Spark, we should make use of our DataFrame friends. Pretraned pipelines allow us to do so with the same function, just adding the target string column. The result here will be a dataframe with many annotation columns.

import spark.implicits._
val data = Seq("hello, this is an example sentence").toDF("mainColumn")
BasicPipeline().annotate(data, "mainColumn").show()

Manipulating pipelines

To add a bit of challenge, the output of the previous DataFrame was in terms of Annotation objects. What if we want to deal with just the resulting annotations? We can use the Finisher annotator, retrieve the BasicPipeline, and add them together in a Spark ML Pipeline. Note BasicPipeline expect the target column to be named "text".

import com.johnsnowlabs.nlp.Finisher

val finisher = new Finisher()
    .setInputCols("token", "normal", "lemma", "pos")

val basicPipeline = BasicPipeline().pretrained()

val pipeline = new Pipeline()

    .transform(data.withColumnRenamed("mainColumn", "text"))

Train your own pipeline

Annotator types

Every annotator has a type. Those annotators that share a type, can be used interchangeably, meaning you could you use any of them when needed. For example, when a token type annotator is required by another annotator, such as a sentiment analysis annotator, you can either provide a normalized token or a lemma, as both are of type token.

Necessary imports

Since version 1.5.0 we are making necessary imports easy to reach, base._ will include general Spark NLP transformers and concepts, while annotator._ will include all annotators that we currently provide. We also need SparkML pipelines.

import com.johnsnowlabs.nlp.base._
import com.johnsnowlabs.nlp.annotator._

DocumentAssembler: Getting data in

In order to get through the NLP process, we need to get raw data annotated. There is a special transformer that does this for us: the DocumentAssembler, it creates the first annotation of type Document which may be used by annotators down the road

val documentAssembler = new DocumentAssembler()

Sentence detection and tokenization

In this quick example, we now proceed to identify the sentences in each of our document lines. SentenceDetector requires a Document annotation, which is provided by the DocumentAssembler output, and it's itself a Document type token. The Tokenizer requires a Document annotation type, meaning it works both with DocumentAssembler or SentenceDetector output, in here, we use the sentence output.

val sentenceDetector = new SentenceDetector()

val regexTokenizer = new Tokenizer()

Using Pipelines

Now we want to put all this together and retrieve the results, we use a Pipeline for this. We also include another special transformer, called Finisher to show tokens in a human language. We use an emptyDataFrame in fit() since none of the pipeline stages have a training stage.

val testData = Seq("Lorem ipsum dolor sit amet, " +
    "consectetur adipiscing elit, sed do eiusmod tempor " +
    "incididunt ut labore et dolore magna aliqua.").toDF("text")

val finisher = new Finisher()

val pipeline = new Pipeline()


Where to go next?

Documentation and reference

Detailed information about Spark-NLP concepts, annotators and more may be found HERE

More examples in Scala and Python

We are working on examples to show you how the library may be used in different scenarios, take a look HERE