sparknlp_jsl.annotator.ner.medical_ner_dl_graph_checker#

Classes#

class MedicalNerDLGraphChecker#

Bases: sparknlp.annotator.NerDLGraphChecker

Checks whether a suitable MedicalNerApproach graph is available for the given training dataset, before any computations/training is done. This annotator is useful for custom training cases, where specialized graphs are needed.

Important: This annotator should be used or positioned before any embedding or MedicalNerApproach annotators in the pipeline and will process the whole dataset to extract the required graph parameters.

This annotator requires a dataset with at least two columns: one with tokens and one with the labels. In addition, it requires the used embedding annotator in the pipeline to extract the suitable embedding dimension.

Input Annotation types	Output Annotation type
DOCUMENT, TOKEN	NONE

Parameters:

• inputCols – Column names of input annotations

• labelColumn – Column name for data labels

• embeddingsDim – Dimensionality of embeddings

Examples

>>> import sparknlp
>>> import sparknlp_jsl
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from sparknlp_jsl.annotator import *
>>> from sparknlp.training import CoNLL
>>> from pyspark.ml import Pipeline

This CoNLL dataset already includes a sentence, token and label column with their respective annotator types. If a custom dataset is used, these need to be defined with for example:

>>> conll = CoNLL()
>>> trainingData = conll.readDataset(spark, "PATH/TO/CONLL/FILE")
>>> embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models") \
...     .setInputCols(["sentence", "token"]) \
...     .setOutputCol("embeddings")

This annotatorr requires the data for MedicalNerApproach graphs: text, tokens, labels and the embedding model

>>> nerDLGraphChecker = MedicalNerDLGraphChecker() \
...     .setInputCols(["sentence", "token"]) \
...     .setLabelColumn("label") \
...     .setEmbeddingsModel(embeddings)

>>> nerTagger = MedicalNerApproach() \
...     .setInputCols(["sentence", "token", "embeddings"]) \
...     .setLabelColumn("label") \
...     .setOutputCol("ner") \
...     .setMaxEpochs(1) \
...     .setRandomSeed(0) \
...     .setVerbose(0) \
...     .setEarlyStoppingCriterion(0.50) \
...     .setUseBestModel(True)

>>> pipeline = Pipeline().setStages([nerDLGraphChecker, embeddings, nerTagger])

If we now fit the model with a graph missing, then an exception is raised.

>>> pipelineModel = pipeline.fit(trainingData)

embeddingsDim#

getter_attrs = []#

graphFolder#

inputAnnotatorTypes#

inputCols#

labelColumn#

name = 'MedicalNerDLGraphChecker'#

uid = ''#

clear(param: pyspark.ml.param.Param) → None#

Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters:

extra (dict, optional) – Extra parameters to copy to the new instance

Returns:

Copy of this instance

Return type:

JavaParams

explainParam(param: str | Param) → str#

Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#

Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:

extra (dict, optional) – extra param values

Returns:

merged param map

Return type:

dict

fit(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = ...) → M#

fit(dataset: pyspark.sql.dataframe.DataFrame, params: List[pyspark.ml._typing.ParamMap] | Tuple[pyspark.ml._typing.ParamMap]) → List[M]

Fits a model to the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

• dataset (pyspark.sql.DataFrame) – input dataset.

• params (dict or list or tuple, optional) – an optional param map that overrides embedded params. If a list/tuple of param maps is given, this calls fit on each param map and returns a list of models.

Returns:

fitted model(s)

Return type:

Transformer or a list of Transformer

fitMultiple(dataset: pyspark.sql.dataframe.DataFrame, paramMaps: Sequence[pyspark.ml._typing.ParamMap]) → Iterator[Tuple[int, M]]#

Fits a model to the input dataset for each param map in paramMaps.

New in version 2.3.0.

Parameters:

• dataset (pyspark.sql.DataFrame) – input dataset.

• paramMaps (collections.abc.Sequence) – A Sequence of param maps.

Returns:

A thread safe iterable which contains one model for each param map. Each call to next(modelIterator) will return (index, model) where model was fit using paramMaps[index]. index values may not be sequential.

Return type:

_FitMultipleIterator

getOrDefault(param: str) → Any#

getOrDefault(param: Param[T]) → T

Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getParam(paramName: str) → Param#

Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:

paramName (str) – Name of the parameter

hasDefault(param: str | Param[Any]) → bool#

Checks whether a param has a default value.

hasParam(paramName: str) → bool#

Tests whether this instance contains a param with a given (string) name.

isDefined(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user.

save(path: str) → None#

Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#

Sets a parameter in the embedded param map.

setEmbeddingsDim(value: int)#

Sets Dimensionality of embeddings

Parameters:

value (int) – Dimensionality of embeddings

setEmbeddingsModel(model: sparknlp.common.HasEmbeddingsProperties)#

Get embeddingsDim from a given embeddings model, if possible. Falls back to setEmbeddingsDim if dimension cannot be obtained automatically.

setGraphFolder(p)#

Sets folder path that contain external graph files.

Parameters:

p (str) – Folder path that contain external graph files

setInputCols(*value)#

Sets column names of input annotations.

Parameters:

*value (List[str]) – Input columns for the annotator

setLabelColumn(value)#

Sets name of column for data labels.

Parameters:

value (str) – Column for data labels

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:

paramName (str) – Name of the parameter

write() → JavaMLWriter#

Returns an MLWriter instance for this ML instance.