sparknlp_jsl.annotator.merge.chunk_merge#

Module Contents#

Classes#

ChunkMergeApproach

Merges two chunk columns coming from two annotators(NER, ContextualParser or any other annotator producing

ChunkMergeModel

The model produced by ChunkMergeApproach.

MergeCommonParams

MergeFeatureParams

MergePrioritizationParams

MergeResourceParams

class ChunkMergeApproach#

Bases: sparknlp_jsl.common.AnnotatorApproachInternal, MergeCommonParams, MergePrioritizationParams, MergeResourceParams, sparknlp_jsl.annotator.filtering_params.FilteringParams, sparknlp_jsl.annotator.handle_exception_params.HandleExceptionParams, sparknlp_jsl.annotator.params.reset_sentence_indices_param.ResetSentenceIndicesParam

Merges two chunk columns coming from two annotators(NER, ContextualParser or any other annotator producing chunks). The merger of the two chunk columns is made by selecting one chunk from one of the columns according to certain criteria. The decision on which chunk to select is made according to the chunk indices in the source document. (chunks with longer lengths and highest information will be kept from each source) Labels can be changed by setReplaceDictResource.

Input Annotation types

Output Annotation type

CHUNK,CHUNK

CHUNK
Parameters:

  • mergeOverlapping – whether to merge overlapping matched chunks. Defaults false

  • falsePositivesResource – file with false positive pairs

  • replaceDictResource – replace dictionary pairs

  • chunkPrecedence – Select what is the precedence when two chunks have the same start and end indices. Possible values are [entity|identifier|field]

  • blackList – If defined, list of entities to ignore. The rest will be proccessed.

  • whiteList – If defined, list of entities to process. The rest will be ignored. Do not include IOB prefix on labels.

  • caseSensitive (bool) – Determines whether the definitions of the white listed and black listed entities are case sensitive. Default: True. If the filterValue is ‘entity’, ‘caseSensitive’ is always False.

  • regex (str) – If defined, list of regex to process the chunks (Default: []).

  • criteria (str) – Tag representing what is the criteria to filter the chunks. Possibles values are: - isIn: Filter by the chunk - regex: Filter using a regex

  • filterValue (str) – If equal to “entity”, use the ner label to filter. If set to “result”, use the result attribute of the annotation to filter.

  • entitiesConfidenceResource (str) – Path to a CSV file containing the entity pairs to remove chunks based on the confidance level. The CSV file should have two columns: entity and confidenceThreshold. The entity column should contain the entity name and the confidenceThreshold column should contain the confidence threshold to filter the chunks.

  • entitiesConfidence (dict[str, float]) – Pairs (entity,confidenceThreshold) to filter which have confidence lower than the confidence threshold.

  • resetSentenceIndices (bool) – Whether to reset sentence indices to treat the entire output as if it originates from a single document.

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp_jsl.common import *
>>> from sparknlp.annotator import *
>>> from sparknlp.training import *
>>> import sparknlp_jsl
>>> from sparknlp_jsl.base import *
>>> from sparknlp_jsl.annotator import *
>>> from pyspark.ml import Pipeline
Define a pipeline with 2 different NER models with a ChunkMergeApproach at the end
>>> data = spark.createDataFrame([["A 63-year-old man presents to the hospital ..."]]).toDF("text")
>>> pipeline = Pipeline(stages=[
...  DocumentAssembler().setInputCol("text").setOutputCol("document"),
...  SentenceDetector().setInputCols(["document"]).setOutputCol("sentence"),
...  Tokenizer().setInputCols(["sentence"]).setOutputCol("token"),
...   WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models").setOutputCol("embs"),
...   MedicalNerModel.pretrained("ner_jsl", "en", "clinical/models") \
...     .setInputCols(["sentence", "token", "embs"]).setOutputCol("jsl_ner"),
...  NerConverter().setInputCols(["sentence", "token", "jsl_ner"]).setOutputCol("jsl_ner_chunk"),
...   MedicalNerModel.pretrained("ner_bionlp", "en", "clinical/models") \
...     .setInputCols(["sentence", "token", "embs"]).setOutputCol("bionlp_ner"),
...  NerConverter().setInputCols(["sentence", "token", "bionlp_ner"]) \
...     .setOutputCol("bionlp_ner_chunk"),
...  ChunkMergeApproach().setInputCols(["jsl_ner_chunk", "bionlp_ner_chunk"]).setOutputCol("merged_chunk")
>>> ])
>>> result = pipeline.fit(data).transform(data).cache()
>>> result.selectExpr("explode(merged_chunk) as a") \
...   .selectExpr("a.begin","a.end","a.result as chunk","a.metadata.entity as entity") \
...   .show(5, False)
+-----+---+-----------+---------+
|begin|end|chunk      |entity   |
+-----+---+-----------+---------+
|5    |15 |63-year-old|Age      |
|17   |19 |man        |Gender   |
|64   |72 |recurrent  |Modifier |
|98   |107|cellulitis |Diagnosis|
|110  |119|pneumonias |Diagnosis|
+-----+---+-----------+---------+
blackList#
caseSensitive#
chunkPrecedence#
chunkPrecedenceValuePrioritization#
criteria#
defaultConfidence#
doExceptionHandling#
entitiesConfidenceResource#
falsePositivesResource#
filterValue#
getter_attrs = []#
inputAnnotatorTypes#
inputCols#
lazyAnnotator#
mergeOverlapping#
name = 'ChunkMergeApproach'#
optionalInputAnnotatorTypes = []#
orderingFeatures#
outputAnnotatorType#
outputCol#
regex#
replaceDictResource#
resetSentenceIndices#
selectionStrategy#
skipLPInputColsValidation = True#
uid#
whiteList#
clear(param: pyspark.ml.param.Param) → None#

Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters:

extra (dict, optional) – Extra parameters to copy to the new instance

Returns:

Copy of this instance

Return type:

JavaParams

explainParam(param: str | Param) → str#

Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#

Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:

extra (dict, optional) – extra param values

Returns:

merged param map

Return type:

dict

fit(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = ...) → M#
fit(dataset: pyspark.sql.dataframe.DataFrame, params: List[pyspark.ml._typing.ParamMap] | Tuple[pyspark.ml._typing.ParamMap]) → List[M]

Fits a model to the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

  • dataset (pyspark.sql.DataFrame) – input dataset.

  • params (dict or list or tuple, optional) – an optional param map that overrides embedded params. If a list/tuple of param maps is given, this calls fit on each param map and returns a list of models.

Returns:

fitted model(s)

Return type:

Transformer or a list of Transformer

fitMultiple(dataset: pyspark.sql.dataframe.DataFrame, paramMaps: Sequence[pyspark.ml._typing.ParamMap]) → Iterator[Tuple[int, M]]#

Fits a model to the input dataset for each param map in paramMaps.

New in version 2.3.0.

Parameters:

  • dataset (pyspark.sql.DataFrame) – input dataset.

  • paramMaps (collections.abc.Sequence) – A Sequence of param maps.

Returns:

A thread safe iterable which contains one model for each param map. Each call to next(modelIterator) will return (index, model) where model was fit using paramMaps[index]. index values may not be sequential.

Return type:

_FitMultipleIterator

getInputCols()#

Gets current column names of input annotations.

getLazyAnnotator()#

Gets whether Annotator should be evaluated lazily in a RecursivePipeline.

getOrDefault(param: str) → Any#
getOrDefault(param: Param[T]) → T

Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getOutputCol()#

Gets output column name of annotations.

getParam(paramName: str) → Param#

Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:

paramName (str) – Name of the parameter

hasDefault(param: str | Param[Any]) → bool#

Checks whether a param has a default value.

hasParam(paramName: str) → bool#

Tests whether this instance contains a param with a given (string) name.

inputColsValidation(value)#
isDefined(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#

Reads an ML instance from the input path, a shortcut of read().load(path).

classmethod read()#

Returns an MLReader instance for this class.

save(path: str) → None#

Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#

Sets a parameter in the embedded param map.

setBlackList(value)#

Sets If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

Parameters:

value (List[str]) – If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

setCaseSensitive(value)#

Determines whether the definitions of the white listed and black listed entities are case sensitive or not.

Parameters:

value (bool) – Whether white listed and black listed entities are case sensitive or not. Default: True.

setChunkPrecedence(value)#

Sets When ChunkPrecedence ordering feature is used this param contains the comma separated fields in metadata that drive prioritization of overlapping annotations. When used by itself (empty chunkPrecedenceValuePrioritization) annotations will be prioritized based on number of metadata fields present. When used together with chunkPrecedenceValuePrioritization param it will prioritize based on the order of its values.

Parameters:

value (string) – When ChunkPrecedence ordering feature is used this param contains the comma separated fields in metadata that drive prioritization of overlapping annotations. When used by itself (empty chunkPrecedenceValuePrioritization) annotations will be prioritized based on number of metadata fields present. When used together with chunkPrecedenceValuePrioritization param it will prioritize based on the order of its values.

setChunkPrecedenceValuePrioritization(value)#

Sets When ChunkPrecedence ordering feature is used this param contains an Array of comma separated values representing the desired order of prioritization for the VALUES in the metadata fields included from chunkPrecedence.

Parameters:

value (List[str]) – When ChunkPrecedence ordering feature is used this param contains an Array of comma separated values representing the desired order of prioritization for the VALUES in the metadata fields included from chunkPrecedence.

setCriteria(value)#

It is used to compare black and white listed values with the result of the Annotation.

Possible values are the following: ‘isin’, ‘regex’. Default: ‘isin’.

isin : Filter by the chunk regex : Filter by using a regex

Parameters:

value (string) – It is used to compare black and white listed values with the result of the Annotation. Possible values are the following: ‘isin’, ‘regex’. Default: ‘isin’.

setDefaultConfidence(value)#

Sets When ChunkConfidence ordering feature is included and a given annotation does not have any confidence the value of this param will be used.

Parameters:

value (float) – When ChunkConfidence ordering feature is included and a given annotation does not have any confidence the value of this param will be used.

setDenyList(value)#

Sets If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

Parameters:

value (List[str]) – If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

setDoExceptionHandling(value: bool)#

If True, exceptions are handled. If exception causing data is passed to the model, a error annotation is emitted which has the exception message. Processing continues with the next one. This comes with a performance penalty.

Parameters:

value (bool) – If True, exceptions are handled.

setEntitiesConfidence(entitiesConfidence: dict)#

Sets the entitiesConfidence parameter.

Parameters:

entities_confidence (dict[str, float]) – Pairs (entity,confidenceThreshold) to filter which have confidence lower than the confidence threshold.

setEntitiesConfidenceResource(path, read_as=ReadAs.TEXT, options=None)#

Sets the entitiesConfidenceResource parameter.

Parameters:

  • path (str) – Path to csv with entity pairs to remove based on the confidence level

  • read_as (str) – Read file as ‘TEXT’, ‘SPARK’, or ‘BINARY’.

  • options (dict) – Options for reading the file.

setFalsePositivesResource(path, read_as=ReadAs.TEXT, options=None)#

Sets file with false positive pairs

Parameters:

  • path (str) – Path to the external resource

  • read_as (str, optional) – How to read the resource, by default ReadAs.TEXT

  • options (dict, optional) – Options for reading the resource, by default {“format”: “text”}

setFilterValue(value)#

Sets possible values ‘result’ or ‘entity’.

If the value is ‘result’, It filters according to the result of the Annotation. If the value is ‘entity’, It filters according to the entity field in the metadata of the Annotation.

Parameters:

value (string) – possible values are ‘result’ and ‘entity’.

setForceInputTypeValidation(etfm)#
setInputCols(*value)#

Sets column names of input annotations. :param *value: Input columns for the annotator :type *value: str

setLazyAnnotator(value)#

Sets whether Annotator should be evaluated lazily in a RecursivePipeline.

Parameters:

value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline

setMergeOverlapping(value)#

Sets whether to merge overlapping chunks. Defaults to true

Parameters:

value (boolean) – whether to merge overlapping chunks. Defaults to true

setOrderingFeatures(value)#

Sets Array of strings specifying the ordering features to use for overlapping entities. Possible values are ChunkBegin, ChunkLength, ChunkPrecedence, ChunkConfidence.

Parameters:

value (List[str]) – Array of strings specifying the ordering features to use for overlapping entities. Possible values are ChunkBegin, ChunkLength, ChunkPrecedence, ChunkConfidence.

setOutputCol(value)#

Sets output column name of annotations.

Parameters:

value (str) – Name of output column

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:

paramName (str) – Name of the parameter

setRegex(value)#

Sets If defined, list of regex to process the chunks.

Parameters:

value (List[str]) – If defined, list of regex to process the chunks

setReplaceDictResource(path, read_as=ReadAs.TEXT, options={'delimiter': ','})#

Sets replace dictionary pairs

Parameters:

  • path (str) – Path to the external resource

  • read_as (str, optional) – How to read the resource, by default ReadAs.TEXT

  • options (dict, optional) – Options for reading the resource, by default {“format”: “text”}

setResetSentenceIndices(value)#

Set whether to reset sentence indices to treat the entire output as if it originates from a single document.

When set to true, the metadata of each entity will be updated by assigning the sentence key a value of 0, effectively treating the entire output as if it originates from a single document. regardless of the original sentence boundaries. Default: False.

Parameters:

value (bool) – If set to true, sentence indices will be reset to treat the entire output as if it originates from a single document.

setSelectionStrategy(value)#

Sets Whether to select annotations sequentially based on annotation order Sequential or using any other available strategy; currently only Sequential and DiverseLonger are available.

Parameters:

value (string) – Whether to select annotations sequentially based on annotation order Sequential or using any other available strategy; currently only Sequential and DiverseLonger are available.

setWhiteList(value)#

Sets If defined, list of entities to process. The rest will be ignored. Do not include IOB prefix on labels

Parameters:

value (List[str]) – If defined, list of entities to process. The rest will be ignored. Do not include IOB prefix on labels

write() → JavaMLWriter#

Returns an MLWriter instance for this ML instance.

class ChunkMergeModel(classname='com.johnsnowlabs.nlp.annotators.merge.ChunkMergeModel', java_model=None)#

Bases: sparknlp_jsl.common.AnnotatorModelInternal, MergeCommonParams, MergePrioritizationParams, sparknlp_jsl.annotator.filtering_params.FilteringParams, MergeFeatureParams, sparknlp_jsl.annotator.handle_exception_params.HandleExceptionParams, sparknlp_jsl.annotator.params.reset_sentence_indices_param.ResetSentenceIndicesParam

The model produced by ChunkMergeApproach.

Input Annotation types

Output Annotation type

CHUNK,CHUNK

CHUNK
Parameters:

  • mergeOverlapping – whether to merge overlapping matched chunks. Defaults false

  • chunkPrecedence – Select what is the precedence when two chunks have the same start and end indices. Possible values are [entity|identifier|field]

  • blackList – If defined, list of entities to ignore. The rest will be proccessed.

  • whiteList – If defined, list of entities to process. The rest will be ignored. Do not include IOB prefix on labels.

  • caseSensitive (bool) – Determines whether the definitions of the white listed and black listed entities are case sensitive. Default: True. If the filterValue is ‘entity’, ‘caseSensitive’ is always False.

  • regex (str) – If defined, list of regex to process the chunks (Default: []).

  • criteria (str) – Tag representing what is the criteria to filter the chunks. Possibles values are: - isIn: Filter by the chunk - regex: Filter using a regex

  • filterValue (str) – If equal to “entity”, use the ner label to filter. If set to “result”, use the result attribute of the annotation to filter.

  • falsePositives (List[List[str]]) – List of entity pairs that are false positives. If a third value is defined, the pair will be replaced by that value.

  • replaceDict (dict[str, str]) – dictionary with regular expression patterns that match some protected entity

  • resetSentenceIndices (bool) – Whether to reset sentence indices to treat the entire output as if it originates from a single document.

blackList#
caseSensitive#
chunkPrecedence#
chunkPrecedenceValuePrioritization#
criteria#
defaultConfidence#
filterValue#
getter_attrs = []#
inputAnnotatorTypes#
inputCols#
lazyAnnotator#
mergeOverlapping#
name = 'ChunkMergeModel'#
optionalInputAnnotatorTypes = []#
orderingFeatures#
outputAnnotatorType#
outputCol#
regex#
selectionStrategy#
skipLPInputColsValidation = True#
uid#
whiteList#
clear(param: pyspark.ml.param.Param) → None#

Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters:

extra (dict, optional) – Extra parameters to copy to the new instance

Returns:

Copy of this instance

Return type:

JavaParams

explainParam(param: str | Param) → str#

Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#

Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:

extra (dict, optional) – extra param values

Returns:

merged param map

Return type:

dict

getInputCols()#

Gets current column names of input annotations.

getLazyAnnotator()#

Gets whether Annotator should be evaluated lazily in a RecursivePipeline.

getOrDefault(param: str) → Any#
getOrDefault(param: Param[T]) → T

Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getOutputCol()#

Gets output column name of annotations.

getParam(paramName: str) → Param#

Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:

paramName (str) – Name of the parameter

hasDefault(param: str | Param[Any]) → bool#

Checks whether a param has a default value.

hasParam(paramName: str) → bool#

Tests whether this instance contains a param with a given (string) name.

inputColsValidation(value)#
isDefined(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#

Reads an ML instance from the input path, a shortcut of read().load(path).

static pretrained(name, lang='en', remote_loc=None)#

Downloads and loads a pretrained model.

Parameters:

  • name (str, optional) – Name of the pretrained model.

  • lang (str, optional) – Language of the pretrained model, by default “en”

  • remote_loc (str, optional) – Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

The restored model

Return type:

ChunkMergeModel

classmethod read()#

Returns an MLReader instance for this class.

save(path: str) → None#

Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#

Sets a parameter in the embedded param map.

setBlackList(value)#

Sets If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

Parameters:

value (List[str]) – If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

setCaseSensitive(value)#

Determines whether the definitions of the white listed and black listed entities are case sensitive or not.

Parameters:

value (bool) – Whether white listed and black listed entities are case sensitive or not. Default: True.

setChunkPrecedence(value)#

Sets When ChunkPrecedence ordering feature is used this param contains the comma separated fields in metadata that drive prioritization of overlapping annotations. When used by itself (empty chunkPrecedenceValuePrioritization) annotations will be prioritized based on number of metadata fields present. When used together with chunkPrecedenceValuePrioritization param it will prioritize based on the order of its values.

Parameters:

value (string) – When ChunkPrecedence ordering feature is used this param contains the comma separated fields in metadata that drive prioritization of overlapping annotations. When used by itself (empty chunkPrecedenceValuePrioritization) annotations will be prioritized based on number of metadata fields present. When used together with chunkPrecedenceValuePrioritization param it will prioritize based on the order of its values.

setChunkPrecedenceValuePrioritization(value)#

Sets When ChunkPrecedence ordering feature is used this param contains an Array of comma separated values representing the desired order of prioritization for the VALUES in the metadata fields included from chunkPrecedence.

Parameters:

value (List[str]) – When ChunkPrecedence ordering feature is used this param contains an Array of comma separated values representing the desired order of prioritization for the VALUES in the metadata fields included from chunkPrecedence.

setCriteria(value)#

It is used to compare black and white listed values with the result of the Annotation.

Possible values are the following: ‘isin’, ‘regex’. Default: ‘isin’.

isin : Filter by the chunk regex : Filter by using a regex

Parameters:

value (string) – It is used to compare black and white listed values with the result of the Annotation. Possible values are the following: ‘isin’, ‘regex’. Default: ‘isin’.

setDefaultConfidence(value)#

Sets When ChunkConfidence ordering feature is included and a given annotation does not have any confidence the value of this param will be used.

Parameters:

value (float) – When ChunkConfidence ordering feature is included and a given annotation does not have any confidence the value of this param will be used.

setDenyList(value)#

Sets If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

Parameters:

value (List[str]) – If defined, list of entities to ignore. The rest will be processed. Do not include IOB prefix on labels

setFilterValue(value)#

Sets possible values ‘result’ or ‘entity’.

If the value is ‘result’, It filters according to the result of the Annotation. If the value is ‘entity’, It filters according to the entity field in the metadata of the Annotation.

Parameters:

value (string) – possible values are ‘result’ and ‘entity’.

setForceInputTypeValidation(etfm)#
setInputCols(*value)#

Sets column names of input annotations. :param *value: Input columns for the annotator :type *value: str

setLazyAnnotator(value)#

Sets whether Annotator should be evaluated lazily in a RecursivePipeline.

Parameters:

value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline

setMergeOverlapping(value)#

Sets whether to merge overlapping chunks. Defaults to true

Parameters:

value (boolean) – whether to merge overlapping chunks. Defaults to true

setOrderingFeatures(value)#

Sets Array of strings specifying the ordering features to use for overlapping entities. Possible values are ChunkBegin, ChunkLength, ChunkPrecedence, ChunkConfidence.

Parameters:

value (List[str]) – Array of strings specifying the ordering features to use for overlapping entities. Possible values are ChunkBegin, ChunkLength, ChunkPrecedence, ChunkConfidence.

setOutputCol(value)#

Sets output column name of annotations.

Parameters:

value (str) – Name of output column

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:

paramName (str) – Name of the parameter

setParams()#
setRegex(value)#

Sets If defined, list of regex to process the chunks.

Parameters:

value (List[str]) – If defined, list of regex to process the chunks

setSelectionStrategy(value)#

Sets Whether to select annotations sequentially based on annotation order Sequential or using any other available strategy; currently only Sequential and DiverseLonger are available.

Parameters:

value (string) – Whether to select annotations sequentially based on annotation order Sequential or using any other available strategy; currently only Sequential and DiverseLonger are available.

setWhiteList(value)#

Sets If defined, list of entities to process. The rest will be ignored. Do not include IOB prefix on labels

Parameters:

value (List[str]) – If defined, list of entities to process. The rest will be ignored. Do not include IOB prefix on labels

transform(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = None) → pyspark.sql.dataframe.DataFrame#

Transforms the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

  • dataset (pyspark.sql.DataFrame) – input dataset

  • params (dict, optional) – an optional param map that overrides embedded params.

Returns:

transformed dataset

Return type:

pyspark.sql.DataFrame

write() → JavaMLWriter#

Returns an MLWriter instance for this ML instance.

class MergeCommonParams#
mergeOverlapping#
setMergeOverlapping(value)#

Sets whether to merge overlapping chunks. Defaults to true

Parameters:

value (boolean) – whether to merge overlapping chunks. Defaults to true

class MergeFeatureParams#
getReplaceDict()#

Return the dictionary with regular expression patterns that match some protected entity

setFalsePositives(value)#

List of entity pairs that are false positives. If a third value is defined, the pair will be replaced by that value. >>> … .setFalsePositives([[“is trying”, “CHANGE”, “WOW”],[“beautiful thing”, “BLOCK”, “”]])

Parameters:

value (List[List[str]]) –

setReplaceDict(replaceDict)#

dictionary with regular expression patterns that match some protected entity

Parameters:

replaceDict (dict[str, str]) –

class MergePrioritizationParams#
chunkPrecedence#
chunkPrecedenceValuePrioritization#
defaultConfidence#
orderingFeatures#
selectionStrategy#
setChunkPrecedence(value)#

Sets When ChunkPrecedence ordering feature is used this param contains the comma separated fields in metadata that drive prioritization of overlapping annotations. When used by itself (empty chunkPrecedenceValuePrioritization) annotations will be prioritized based on number of metadata fields present. When used together with chunkPrecedenceValuePrioritization param it will prioritize based on the order of its values.

Parameters:

value (string) – When ChunkPrecedence ordering feature is used this param contains the comma separated fields in metadata that drive prioritization of overlapping annotations. When used by itself (empty chunkPrecedenceValuePrioritization) annotations will be prioritized based on number of metadata fields present. When used together with chunkPrecedenceValuePrioritization param it will prioritize based on the order of its values.

setChunkPrecedenceValuePrioritization(value)#

Sets When ChunkPrecedence ordering feature is used this param contains an Array of comma separated values representing the desired order of prioritization for the VALUES in the metadata fields included from chunkPrecedence.

Parameters:

value (List[str]) – When ChunkPrecedence ordering feature is used this param contains an Array of comma separated values representing the desired order of prioritization for the VALUES in the metadata fields included from chunkPrecedence.

setDefaultConfidence(value)#

Sets When ChunkConfidence ordering feature is included and a given annotation does not have any confidence the value of this param will be used.

Parameters:

value (float) – When ChunkConfidence ordering feature is included and a given annotation does not have any confidence the value of this param will be used.

setOrderingFeatures(value)#

Sets Array of strings specifying the ordering features to use for overlapping entities. Possible values are ChunkBegin, ChunkLength, ChunkPrecedence, ChunkConfidence.

Parameters:

value (List[str]) – Array of strings specifying the ordering features to use for overlapping entities. Possible values are ChunkBegin, ChunkLength, ChunkPrecedence, ChunkConfidence.

setSelectionStrategy(value)#

Sets Whether to select annotations sequentially based on annotation order Sequential or using any other available strategy; currently only Sequential and DiverseLonger are available.

Parameters:

value (string) – Whether to select annotations sequentially based on annotation order Sequential or using any other available strategy; currently only Sequential and DiverseLonger are available.

class MergeResourceParams#
entitiesConfidenceResource#
falsePositivesResource#
replaceDictResource#
setEntitiesConfidenceResource(path, read_as=ReadAs.TEXT, options=None)#

Sets the entitiesConfidenceResource parameter.

Parameters:

  • path (str) – Path to csv with entity pairs to remove based on the confidence level

  • read_as (str) – Read file as ‘TEXT’, ‘SPARK’, or ‘BINARY’.

  • options (dict) – Options for reading the file.

setFalsePositivesResource(path, read_as=ReadAs.TEXT, options=None)#

Sets file with false positive pairs

Parameters:

  • path (str) – Path to the external resource

  • read_as (str, optional) – How to read the resource, by default ReadAs.TEXT

  • options (dict, optional) – Options for reading the resource, by default {“format”: “text”}

setReplaceDictResource(path, read_as=ReadAs.TEXT, options={'delimiter': ','})#

Sets replace dictionary pairs

Parameters:

  • path (str) – Path to the external resource

  • read_as (str, optional) – How to read the resource, by default ReadAs.TEXT

  • options (dict, optional) – Options for reading the resource, by default {“format”: “text”}