`sparknlp_jsl.annotator.er.entity_ruler_internal`#

Contains classes for the EntityRulerInternal.

Module Contents#

Classes#

`EntityRulerInternalApproach`	Fits an Annotator to match exact strings or regex patterns provided in a
`EntityRulerInternalModel`	Instantiated model of the EntityRulerInternalApproach.

class EntityRulerInternalApproach#

Bases: sparknlp_jsl.common.AnnotatorApproachInternal, sparknlp_jsl.common.HasStorage

Fits an Annotator to match exact strings or regex patterns provided in a file against a Document and assigns them an named entity. The definitions can contain any number of named entities.

There are multiple ways and formats to set the extraction resource. It is possible to set it either as a “JSON”, “JSONL” or “CSV” file. A path to the file needs to be provided to setPatternsResource. The file format needs to be set as the “format” field in the option parameter map and depending on the file type, additional parameters might need to be set.

If the file is in a JSON format, then the rule definitions need to be given in a list with the fields “id”, “label” and “patterns”:

 [
    {
      "id": "person-regex",
      "label": "PERSON",
      "patterns": ["\w+\s\w+", "\w+-\w+"]
    },
    {
      "id": "locations-words",
      "label": "LOCATION",
      "patterns": ["Winterfell"]
    }
]

The same fields also apply to a file in the JSONL format:

{"id": "names-with-j", "label": "PERSON", "patterns": ["Jon", "John", "John Snow"]}
{"id": "names-with-s", "label": "PERSON", "patterns": ["Stark", "Snow"]}
{"id": "names-with-e", "label": "PERSON", "patterns": ["Eddard", "Eddard Stark"]}

In order to use a CSV file, an additional parameter “delimiter” needs to be set. In this case, the delimiter might be set by using .setPatternsResource("patterns.csv", ReadAs.TEXT, {"format": "csv", "delimiter": "|")}):

PERSON|Jon
PERSON|John
PERSON|John Snow
LOCATION|Winterfell

Input Annotation types	Output Annotation type
`DOCUMENT, TOKEN`	`CHUNK`

Parameters:

patternsResource – Resource in JSON or CSV format to map entities to patterns
useStorage – Whether to use RocksDB storage to serialize patterns

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from sparknlp.common import *
>>> from pyspark.ml import Pipeline

In this example, the entities file as the form of:

PERSON|Jon
PERSON|John
PERSON|John Snow
LOCATION|Winterfell

where each line represents an entity and the associated string delimited by “|”.

>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("document")
>>> tokenizer = Tokenizer() \
...     .setInputCols(["document"]) \
...     .setOutputCol("token")
>>> entityRuler = EntityRulerInternalApproach() \
...     .setInputCols(["document", "token"]) \
...     .setOutputCol("entities") \
...     .setPatternsResource(
...       "patterns.csv",
...       ReadAs.TEXT,
...       {"format": "csv", "delimiter": "\\|"}
...     )
>>> pipeline = Pipeline().setStages([
...     documentAssembler,
...     tokenizer,
...     entityRuler
... ])
>>> data = spark.createDataFrame([["Jon Snow wants to be lord of Winterfell."]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.selectExpr("explode(entities)").show(truncate=False)
+--------------------------------------------------------------------+
|col                                                                 |
+--------------------------------------------------------------------+
|[chunk, 0, 2, Jon, [entity -> PERSON, sentence -> 0], []]           |
|[chunk, 29, 38, Winterfell, [entity -> LOCATION, sentence -> 0], []]|
+--------------------------------------------------------------------+

alphabet#

caseSensitive#

enableInMemoryStorage#

getter_attrs = []#

includeStorage#

inputAnnotatorTypes#

inputCols#

lazyAnnotator#

name = 'EntityRulerInternalApproach'#

optionalInputAnnotatorTypes#

outputAnnotatorType#

outputCol#

patternsResource#

sentenceMatch#

skipLPInputColsValidation = True#

storagePath#

storageRef#

uid#

useStorage#

clear(param: pyspark.ml.param.Param) → None#: Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters:: extra (dict, optional) – Extra parameters to copy to the new instance
Returns:: Copy of this instance
Return type:: JavaParams

explainParam(param: str | Param) → str#: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:: extra (dict, optional) – extra param values
Returns:: merged param map
Return type:: dict

fit(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = ...) → M#

fit(dataset: pyspark.sql.dataframe.DataFrame, params: List[pyspark.ml._typing.ParamMap] | Tuple[pyspark.ml._typing.ParamMap]) → List[M]

Fits a model to the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

dataset (pyspark.sql.DataFrame) – input dataset.
params (dict or list or tuple, optional) – an optional param map that overrides embedded params. If a list/tuple of param maps is given, this calls fit on each param map and returns a list of models.

Returns:

fitted model(s)

Return type:

Transformer or a list of Transformer

fitMultiple(dataset: pyspark.sql.dataframe.DataFrame, paramMaps: Sequence[pyspark.ml._typing.ParamMap]) → Iterator[Tuple[int, M]]#

Fits a model to the input dataset for each param map in paramMaps.

New in version 2.3.0.

Parameters:

dataset (pyspark.sql.DataFrame) – input dataset.
paramMaps (collections.abc.Sequence) – A Sequence of param maps.

Returns:

A thread safe iterable which contains one model for each param map. Each call to next(modelIterator) will return (index, model) where model was fit using paramMaps[index]. index values may not be sequential.

Return type:

_FitMultipleIterator

getCaseSensitive()#

Gets whether to ignore case in tokens for embeddings matching.

Returns:: Whether to ignore case in tokens for embeddings matching
Return type:: bool

getEnableInMemoryStorage()#

getIncludeStorage()#

Gets whether to include indexed storage in trained model.

Returns:: Whether to include indexed storage in trained model
Return type:: bool

getInputCols()#: Gets current column names of input annotations.

getLazyAnnotator()#: Gets whether Annotator should be evaluated lazily in a RecursivePipeline.

getOrDefault(param: str) → Any#
getOrDefault(param: Param[T]) → T: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getOutputCol()#: Gets output column name of annotations.

getParam(paramName: str) → Param#: Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

getStoragePath()#

Gets path to file.

Returns:: path to file
Return type:: str

getStorageRef()#

Gets unique reference name for identification.

Returns:: Unique reference name for identification
Return type:: str

hasDefault(param: str | Param[Any]) → bool#: Checks whether a param has a default value.

hasParam(paramName: str) → bool#: Tests whether this instance contains a param with a given (string) name.

inputColsValidation(value)#

isDefined(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#: Reads an ML instance from the input path, a shortcut of read().load(path).

classmethod read()#: Returns an MLReader instance for this class.

save(path: str) → None#: Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#: Sets a parameter in the embedded param map.

setAlphabetResource(path)#

Alphabet Resource (a simple plain text with all language characters)

Parameters:: path (str) – Path to the resource

setCaseSensitive(value)#

Sets whether to ignore case in tokens for embeddings matching.

Parameters:: value (bool) – Whether to ignore case in tokens for embeddings matching

setEnableInMemoryStorage(value)#

Sets whether to load whole indexed storage in memory (in-memory lookup)

Parameters:: value (bool) – Whether to load whole indexed storage in memory (in-memory lookup)

setForceInputTypeValidation(etfm)#

setIncludeStorage(value)#

Sets whether to include indexed storage in trained model.

Parameters:: value (bool) – Whether to include indexed storage in trained model

setInputCols(*value)#

Sets column names of input annotations.

Parameters:: *value (List[str]) – Input columns for the annotator

setLazyAnnotator(value)#

Sets whether Annotator should be evaluated lazily in a RecursivePipeline.

Parameters:: value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline

setOutputCol(value)#

Sets output column name of annotations.

Parameters:: value (str) – Name of output column

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

setPatternsResource(path, read_as=ReadAs.TEXT, options={'format': 'JSON'})#

Sets Resource in JSON or CSV format to map entities to patterns.

Parameters:

path (str) – Path to the resource
read_as (str, optional) – How to interpret the resource, by default ReadAs.TEXT
options (dict, optional) – Options for parsing the resource, by default {“format”: “JSON”}

setSentenceMatch(value)#

Sets whether to find match at sentence level.

Parameters:: value (bool) – True: sentence level. False: token level

setStoragePath(path, read_as)#

Sets path to file.

Parameters:

path (str) – Path to file
read_as (str) – How to interpret the file

Notes

See ReadAs for reading options.

setStorageRef(value)#

Sets unique reference name for identification.

Parameters:: value (str) – Unique reference name for identification

setUseStorage(value)#

Sets whether to use RocksDB storage to serialize patterns.

Parameters:: value (bool) – Whether to use RocksDB storage to serialize patterns.

write() → JavaMLWriter#: Returns an MLWriter instance for this ML instance.

class EntityRulerInternalModel(classname='com.johnsnowlabs.nlp.annotators.er.EntityRulerInternalModel', java_model=None)#

Bases: sparknlp_jsl.common.AnnotatorModelInternal, sparknlp_jsl.common.HasStorageModel

Instantiated model of the EntityRulerInternalApproach. For usage and examples see the documentation of the main class.

Input Annotation types	Output Annotation type
`DOCUMENT, TOKEN`	`CHUNK`

caseSensitive#

database = ['ENTITY_PATTERNS']#

getter_attrs = []#

inputAnnotatorTypes#

inputCols#

lazyAnnotator#

name = 'EntityRulerInternalModel'#

optionalInputAnnotatorTypes#

outputAnnotatorType#

outputCol#

skipLPInputColsValidation = True#

storageRef#

uid#

clear(param: pyspark.ml.param.Param) → None#: Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Parameters:: extra (dict, optional) – Extra parameters to copy to the new instance
Returns:: Copy of this instance
Return type:: JavaParams

explainParam(param: str | Param) → str#: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Parameters:: extra (dict, optional) – extra param values
Returns:: merged param map
Return type:: dict

getCaseSensitive()#

Gets whether to ignore case in tokens for embeddings matching.

Returns:: Whether to ignore case in tokens for embeddings matching
Return type:: bool

getInputCols()#: Gets current column names of input annotations.

getLazyAnnotator()#: Gets whether Annotator should be evaluated lazily in a RecursivePipeline.

getOrDefault(param: str) → Any#
getOrDefault(param: Param[T]) → T: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getOutputCol()#: Gets output column name of annotations.

getParam(paramName: str) → Param#: Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

getStorageRef()#

Gets unique reference name for identification.

Returns:: Unique reference name for identification
Return type:: str

hasDefault(param: str | Param[Any]) → bool#: Checks whether a param has a default value.

hasParam(paramName: str) → bool#: Tests whether this instance contains a param with a given (string) name.

inputColsValidation(value)#

isDefined(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#: Reads an ML instance from the input path, a shortcut of read().load(path).

static loadStorage(path, spark, storage_ref)#

static loadStorages(path, spark, storage_ref, databases)#

static pretrained(name, lang='en', remote_loc=None)#

classmethod read()#: Returns an MLReader instance for this class.

save(path: str) → None#: Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

saveStorage(path, spark)#

Saves the current model to storage.

Parameters:

path (str) – Path for saving the model.
spark (pyspark.sql.SparkSession) – The current SparkSession

set(param: Param, value: Any) → None#: Sets a parameter in the embedded param map.

setCaseSensitive(value)#

Sets whether to ignore case in tokens for embeddings matching.

Parameters:: value (bool) – Whether to ignore case in tokens for embeddings matching

setForceInputTypeValidation(etfm)#

setInputCols(*value)#

Sets column names of input annotations.

Parameters:: *value (List[str]) – Input columns for the annotator

setLazyAnnotator(value)#

Sets whether Annotator should be evaluated lazily in a RecursivePipeline.

Parameters:: value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline

setOutputCol(value)#

Sets output column name of annotations.

Parameters:: value (str) – Name of output column

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

setParams()#

setStorageRef(value)#

Sets unique reference name for identification.

Parameters:: value (str) – Unique reference name for identification

transform(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = None) → pyspark.sql.dataframe.DataFrame#

Transforms the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

dataset (pyspark.sql.DataFrame) – input dataset
params (dict, optional) – an optional param map that overrides embedded params.

Returns:

transformed dataset

Return type:

pyspark.sql.DataFrame

write() → JavaMLWriter#: Returns an MLWriter instance for this ML instance.

sparknlp_jsl.annotator.er.entity_ruler_internal#

Module Contents#

Classes#

`sparknlp_jsl.annotator.er.entity_ruler_internal`#