`sparknlp_jsl.annotator.flattener`#

Module Contents#

Classes#

Flattener

Converts annotation results into a format that easier to use.

class Flattener#

Bases: sparknlp_jsl.annotator.AnnotatorTransformer

Converts annotation results into a format that easier to use.

The Flattener produces a DataFrame with flattened and exploded columns containing annotation results, making it easier to interpret and analyze the information. It is particularly useful for extracting and organizing the results obtained from Spark NLP Pipelines.

Input Annotation types	Output Annotation type
`ANY`	`NONE`

Parameters:

inputCols – Input annotations
cleanAnnotations – Whether to remove annotation columns, by default True
explodeSelectedFields – Dict of input columns to their corresponding selected fields
flattenExplodedColumns – Whether to flatten exploded columns(default : true)
orderByColumn – Specify the column by which the DataFrame should be ordered.
orderDescending – specifying whether to order the DataFrame in descending order.(default : true)

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from sparknlp.pretrained import PretrainedPipeline
>>> data = spark.createDataFrame([["He is an elderly gentleman in no acute distress."]]).toDF("text")

>>> documentAssembler = DocumentAssembler().setInputCol("text").setOutputCol("document")
>>> sentenceDetector = SentenceDetector().setInputCols(["document"]).setOutputCol("sentence")
>>> tokenizer = Tokenizer().setInputCols(["sentence"]).setOutputCol("token")
>>> word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
>>>    .setInputCols(["sentence", "token"]).setOutputCol("embeddings")
>>> clinical_ner = MedicalNerModel.pretrained("ner_jsl", "en", "clinical/models")
>>>    .setInputCols(["sentence", "token", "embeddings"]).setOutputCol("ner")
>>> ner_converter = NerConverterInternal().setInputCols(["sentence", "token", "ner"]).setOutputCol("ner_chunk")     >>> clinical_assertion = AssertionDLModel.pretrained("assertion_jsl_augmented", "en", "clinical/models")
>>>    .setInputCols(["sentence", "ner_chunk", "embeddings"]).setOutputCol("assertion")     >>>    .setEntityAssertionCaseSensitive(False)

>>> nlpPipeline = Pipeline(stages=[documentAssembler, sentenceDetector,tokenizer, word_embeddings,clinical_ner, ner_converter,clinical_assertion, finisher) ])
>>> flattener = Flattener()
>>>        .setInputCols("sentence", "ner_chunk", "assertion")     >>>        .setExplodeSelectedFields({"ner_chunk": ["result", "metadata.entity"],
>>>                                   "assertion": ["result", "metadata.confidence"]})     >>>        .setOrderByColumn("assertion_metadata_confidence")
>>> explainResult = pipeline.transform(data)

>>> model = nlpPipeline.fit(data).transform(data)
>>> model.select("finished_ner_chunk_exploded").show(truncate=False)

Show results.

ner_chunk_result	ner_chunk_metadata_entity	assertion_result	assertion_metadata_confidence
elderly gentleman He acute distress	Age Gender Gender Modifier Symptom	Present Absent SomeoneElse Absent Absent	0.9885 0.9976 0.9994 1.0 1.0

cleanAnnotations#

flattenExplodedColumns#

getter_attrs = []#

inputCols#

keepOriginalColumns#

kwargs#

name = 'Flattener'#

orderByColumn#

orderDescending#

outputAnnotatorType = None#

uid#

clear(param: pyspark.ml.param.Param) → None#: Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters:: extra (dict, optional) – Extra parameters to copy to the new instance
Returns:: Copy of this instance
Return type:: JavaParams

explainParam(param: str | Param) → str#: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:: extra (dict, optional) – extra param values
Returns:: merged param map
Return type:: dict

getInputCols()#: Gets input columns name of annotations.

getOrDefault(param: str) → Any#
getOrDefault(param: Param[T]) → T: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getParam(paramName: str) → Param#: Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

hasDefault(param: str | Param[Any]) → bool#: Checks whether a param has a default value.

hasParam(paramName: str) → bool#: Tests whether this instance contains a param with a given (string) name.

isDefined(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#: Reads an ML instance from the input path, a shortcut of read().load(path).

classmethod read()#: Returns an MLReader instance for this class.

save(path: str) → None#: Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#: Sets a parameter in the embedded param map.

setCleanAnnotations(value)#

Sets whether to remove annotation columns, by default True.

Parameters:: value (bool) – Whether to remove annotation columns

setExplodeSelectedFields(value)#

Sets a dict of input columns to their corresponding selected fields.

When set, the transformation returns an exploded column for each specified field containing annotation data. This allows you to choose and explode only the desired fields.

If explodeSelectedFields is not set (default), the transformation will return all information for the specified columns.

Alias can be given with as

Parameters:: value (dict) – Map of input columns to their corresponding selected fields

setFlattenExplodedColumns(value)#

Sets whether to flatten exploded columns. When `true`(the default), the transformation returns a flattened and exploded columns containing annotation data, providing a comprehensive view of the annotated information.

When set to false , the transformation returns exploded columns without flattening

Parameters:: value (bool) – whether to flatten exploded columns

setInputCols(*value)#

Sets column names of input annotations.: If explodeSelectedFields is not set (default), the transformation will return all information for the specified columns.

Parameters:: *value (str) – Input columns for the annotator

setKeepOriginalColumns(value)#

Sets array of column names that should be kept in the DataFrame: after the flattening process. These columns will not be affected by the flattening operation and will be included in the final output as they are.

Parameters:: value (list[str]) – Array of column names that should be kept in the DataFrame after the flattening process.

setOrderByColumn(value)#

Sets the column by which the DataFrame should be ordered.: flattenExplodedColumns must be true for ordering

Parameters:: value (bool) – the column by which the DataFrame should be ordered.

setOrderDescending(value)#

Sets whether to order the DataFrame in descending order.(defaulttrue): flattenExplodedColumns must be true for ordering

Parameters:: value (bool) – whether to order the DataFrame in descending order.(default : true)

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

setParams()#

transform(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = None) → pyspark.sql.dataframe.DataFrame#

Transforms the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

dataset (pyspark.sql.DataFrame) – input dataset
params (dict, optional) – an optional param map that overrides embedded params.

Returns:

transformed dataset

Return type:

pyspark.sql.DataFrame

write() → JavaMLWriter#: Returns an MLWriter instance for this ML instance.

sparknlp_jsl.annotator.flattener#

Module Contents#

Classes#

`sparknlp_jsl.annotator.flattener`#