The predict() function

predict() expects either a column named ‘text’ in the dataframe passed to it, or alternatively it will assume the first column of the dataframe passed to it as the column it should predict for.

Predict method Parameters

Output metadata

The NLP predict method has a boolean metadata parameter.
When it is set to True, it output the confidence and additional metadata for each prediction. Its default value is False.

nlp.load('lang').predict('What a wonderful day!')

Output Level parameter

predict() defines 4 output levels for the generated predictions.
The output levels define how granular the predictions and outputs will be.
Depending on your goal, may need to be output level should be adjusted.

1. Token level: Outputs one row for every token in the input. One to many mapping.
2. Chunk level: Outputs one row for every chunk in the input. One to many mapping.
3. Sentence level: Outputs one row for every sentence the input. One to many mapping.
4. Relation level output: Outputs one row for every relation predicted, i.e. . One to many.
5. Document level output: Outputs one row for every document in the input. One to one mapping.

predict() will try to infer the most useful output level automatically if an output level is not specified.
The inferred output level will usually define the last element of the pipeline.

Take a look at the different output levels Demo which goes over all the output levels.

Document output level example

Every row in the input data frame will be mapped to one row in the output dataframe.

# outputs 1 row for 1 input document
nlp.load('sentiment').predict(['I love data science! It is so much fun! It can also be quite helpful to people.', 'I love the city New-York'], output_level='document')

Sentence output level example

Every sentence in each row becomes a new row in the output dataframe.

# will detect the 2 sentences and output 2 rows, one for each of the sentences.
nlp.load('sentiment').predict(['I love data science! It is so much fun! It can also be quite helpful to people.', 'I love the city New-York'], output_level='sentence')

Chunk output level example

Every chunk in each input row becomes a new row in the output dataframe. This is useful for components like the Named Entity Resolver. By setting output level to chunk, you will ensure ever Named Entity becomes one row in your datset. Named Entities are chunks.

# 'New York' is a Chunk. A chunk is an object that consists of multiple tokens, but it's not a sentence.
nlp.load('ner').predict(['Angela Merkel and Donald Trump dont share many opinions', "Ashley wants to visit the Brandenburger Tor in Berlin"], output_level='chunk',)

Token output level example

Every token in each input row becomes a new row in the output dataframe.

# Every token in our sentence will become a row
nlp.load('sentiment').predict(['I love data science! It is so much fun! It can also be quite helpful to people.', 'I love the city New-York'], output_level='token')

Output positions parameter

By setting output_positions=True, the Dataframe generated will contain additional columns which describe the beginning and end of each feature inside of the original document. These additional _begining and _end columns let you infer the piece of the original input string that has been used to generate the output.

• If output level is set to a different output level than some features output level, the resulting features will be inside of lists
• If output level is set to the same output level as some feature, the generated positional features will be single integers
• positional :
• For token based components the positional features refer to the beginning and the end of the token inside the original document the text originates from.
• For sentence based components like sentence embeddings and different sentence classifiers the output of positional will describe the beginning and the end of the sentence that was used to generate the output.

nlp.load('sentiment').predict('I love data science!', output_level='token', output_positions=True)

Row origin inference for one to many mappings

predict() will recycle the Pandas index from the input Dataframe.
The index is useful if one row is mapped to many rows during prediction.
The new rows which are generated from the input row will all have the same index as the original source row.
I.e. if one sentence row gets split into many token rows, each token row will have the same index as the sentence row.

NaN Handling

• Every NaN value is converted to a Python None variable which is reflected in the final dataframe
• If a column contains only NaN or None, it will be dropped

Memory optimization recommendations

Instead of passing your entire Pandas Dataframe to predict() you can pass only the columns which you need for later tasks.
This saves memory and computation time and can be achieved like in the following example, which assumes latitude and longitude are irrelevant for later tasks.

from johnsnowlabs import nlp
import pandas as pd
data = {
   'tweet': ['@CKL-IT the john snow labs library is awesome!', '@MaziyarPanahi johnsnowlabs library is pretty cool', '@JohnSnowLabs Try out the johnsnowlabs library!'],
   'tweet_location': ['Berlin', 'Paris', 'United States'],
   'tweet_lattitude' : ['52.55035', '48.858093', '40.689247'],
   'tweet_longtitude' : ['13.39139', '2.294694','-74.044502']
   }

  
text_df = pd.DataFrame(data)
nlp.load('sentiment').predict(text_df[['tweet','tweet_location']])

Supported data types

predict() supports all of the common Python data types and formats

• Pandas Dataframes
• Spark Dataframes
• Modin with Dask backend
• Modin with Ray backend
• 1-D Numpy arrays of Strings
• Strings
• Arrays of Strings

Single strings

from johnsnowlabs import nlp 
nlp.load('sentiment').predict('This is just one string')

Lists of strings

from johnsnowlabs import nlp
nlp.load('sentiment').predict(['This is an array', ' Of strings!'])

Pandas Dataframe

One column must be named text and of object/string type or the first column will be used instead if no column named ‘text’ exists note : Passing the entire dataframe with additional features to the predict() method is very memory intensive.
It is recommended to only pass the columns required for further downstream tasks to the predict() method.

from johnsnowlabs import nlp
import pandas as pd
data = {"text": ['This day sucks', 'I love this day', 'I dont like Sami']}
text_df = pd.DataFrame(data)
nlp.load('sentiment').predict(text_df)

Pandas Series

One column must be named text and of object/string type
note : This way is the most memory efficient way

from johnsnowlabs import nlp
import pandas as pd
data = {"text": ['This day sucks', 'I love this day', 'I dont like Sami']}
text_df = pd.DataFrame(data)
nlp.load('sentiment').predict(text_df['text'])

Spark Dataframe

One column must be named text and of string type or the first column will be used instead if no column named ‘text’ exists

from johnsnowlabs import nlp
import pandas as pd
data = {"text": ['This day sucks', 'I love this day', 'I dont like Sami']}
text_pdf = pd.DataFrame(data)
text_sdf = nlp.spark.createDataFrame(text_pdf)
nlp.load('sentiment').predict(text_sdf)

Modin Dataframe

Supports Ray Dask backends
One column must be named text and of string type or the first column will be used instead if no column named ‘text’ exists

from johnsnowlabs import nlp
import modin.pandas as pd
data = {"text": ['This day sucks', 'I love this day', 'I don't like Sami']}
text_pdf = pd.DataFrame(data)
nlp.load('sentiment').predict(text_pdf)