`sparknlp_jsl.annotator.re.relation_ner_chunk_filter`#

Module Contents#

Classes#

RENerChunksFilter

Filters entities' dependency relations.

class RENerChunksFilter(classname='com.johnsnowlabs.nlp.annotators.re.RENerChunksFilter', java_model=None)#

Bases: sparknlp_jsl.common.AnnotatorModel

Filters entities’ dependency relations.

The annotator filters desired relation pairs (defined by the parameter realtionPairs), and store those on the output column.

Filtering the possible relations can be useful to perform additional analysis for a specific use case (e.g., checking adverse drug reactions and drug realations), which can be the input for further analysis using a pretrained RelationExtractionDLModel.

For example, the [ner_clinical](https://nlp.johnsnowlabs.com/2021/03/31/ner_clinical_en.html) NER model can identify PROBLEM, TEST, and TREATMENT entities. By using the RENerChunksFilter, one can filter only the relations between PROBLEM and TREATMENT entities only, removing any relation between the other entities, to further analyze the associations between clinical problems and treatments.

Input Annotation types	Output Annotation type
`CHUNK,DEPENDENCY`	`CHUNK`

Parameters:

relationPairs – List of valid relations to filter. Each relation is a dash-separated pair of named entities. E.g., [“problem-treatment”]. If not set, all relations between the entities will be considered.
relationPairsCaseSensitive – Determines whether relation pairs are case sensitive
maxSyntacticDistance – Maximum syntactic distance between a pair of named entities to consider them as a relation. Increase this value if you want to consider relations between entities that are far away, but be careful as it may add false positives (Default: 0).
docLevelRelations – Whether to include relations between entities from different sentences (Default: False).
directionSensitive – Specify direction sensitivity in processing relation pairs Default is ‘false’.
filterByTokenDistance – Filtering criterion based on number of token between entities (Default: 0).

Examples

>>> documenter = DocumentAssembler()\
...   .setInputCol("text")\
...   .setOutputCol("document")
...
>>> sentencer = SentenceDetector()\
...   .setInputCols(["document"])\
...   .setOutputCol("sentences")
...
>>> tokenizer = Tokenizer()\
...   .setInputCols(["sentences"])\
...   .setOutputCol("tokens")
...
>>> words_embedder = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\
...   .setInputCols(["sentences", "tokens"])\
...   .setOutputCol("embeddings")
...
>>> pos_tagger = PerceptronModel.pretrained("pos_clinical", "en", "clinical/models")\
...   .setInputCols(["sentences", "tokens"])\
...   .setOutputCol("pos_tags")
...
>>> dependency_parser = DependencyParserModel.pretrained("dependency_conllu", "en")\
...   .setInputCols(["sentences", "pos_tags", "tokens"])\
...   .setOutputCol("dependencies")
...
>>> clinical_ner_tagger = MedicalNerModel.pretrained("jsl_ner_wip_greedy_clinical","en","clinical/models")\
...   .setInputCols(["sentences", "tokens", "embeddings"])\
...   .setOutputCol("ner_tags")
...
>>> ner_chunker = NerConverter()\
...   .setInputCols(["sentences", "tokens", "ner_tags"])\
...   .setOutputCol("ner_chunks")
...
... # Define the relation pairs and the filter
>>> relationPairs = [
...   "direction-external_body_part_or_region",
...   "external_body_part_or_region-direction",
...   "direction-internal_organ_or_component",
...   "internal_organ_or_component-direction"
... ]
...
>>> re_ner_chunk_filter = RENerChunksFilter()\
...   .setInputCols(["ner_chunks", "dependencies"])\
...   .setOutputCol("re_ner_chunks")\
...   .setMaxSyntacticDistance(4)\
...   .setRelationPairs(["internal_organ_or_component-direction"])
...
>>> trained_pipeline = Pipeline(stages=[
...   documenter,
...   sentencer,
...   tokenizer,
...   words_embedder,
...   pos_tagger,
...   clinical_ner_tagger,
...   ner_chunker,
...   dependency_parser,
...   re_ner_chunk_filter
... ])
...
>>> data = spark.cre>>>DataFrame([["MRI demonstrated infarction in the upper brain stem , left cerebellum and  right basil ganglia"]]).toDF("text")
>>> result = trained_pipeline.fit(data).transform(data)
...
... # Show results
>>> result.selectExpr("explode(re_ner_chunks) as re_chunks") \
...   .selectExpr("re_chunks.begin", "re_chunks.result", "re_chunks.metadata.entity", "re_chunks.metadata.paired_to") \
...   .show(6, truncate=False)
+-----+-------------+---------------------------+---------+
|begin|result       |entity                     |paired_to|
+-----+-------------+---------------------------+---------+
|35   |upper        |Direction                  |41       |
|41   |brain stem   |Internal_organ_or_component|35       |
|35   |upper        |Direction                  |59       |
|59   |cerebellum   |Internal_organ_or_component|35       |
|35   |upper        |Direction                  |81       |
|81   |basil ganglia|Internal_organ_or_component|35       |
+-----+-------------+---------------------------+---------+
+---------------------------------------------------------+

directionSensitive#

docLevelRelations#

filterByTokenDistance#

getter_attrs = []#

inputAnnotatorTypes#

inputCols#

lazyAnnotator#

maxSyntacticDistance#

name = 'RENerChunksFilter'#

optionalInputAnnotatorTypes = []#

outputAnnotatorType = 'chunk'#

outputCol#

relationPairs#

relationPairsCaseSensitive#

uid = ''#

clear(param: pyspark.ml.param.Param) → None#: Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters:: extra (dict, optional) – Extra parameters to copy to the new instance
Returns:: Copy of this instance
Return type:: JavaParams

explainParam(param: str | Param) → str#: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:: extra (dict, optional) – extra param values
Returns:: merged param map
Return type:: dict

getInputCols()#: Gets current column names of input annotations.

getLazyAnnotator()#: Gets whether Annotator should be evaluated lazily in a RecursivePipeline.

getOrDefault(param: str) → Any#
getOrDefault(param: Param[T]) → T: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getOutputCol()#: Gets output column name of annotations.

getParam(paramName: str) → Param#: Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

hasDefault(param: str | Param[Any]) → bool#: Checks whether a param has a default value.

hasParam(paramName: str) → bool#: Tests whether this instance contains a param with a given (string) name.

inputColsValidation(value)#

isDefined(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#: Reads an ML instance from the input path, a shortcut of read().load(path).

classmethod read()#: Returns an MLReader instance for this class.

save(path: str) → None#: Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#: Sets a parameter in the embedded param map.

setDirectionSensitive(value: bool)#

Sets whether to sensitive for direction in processing relation pairs. If it is true, only relations in the form of “ENTITY1-ENTITY2” will be considered, If it is false, both “ENTITY1-ENTITY2” and “ENTITY2-ENTITY1” relations will be considered,

Parameters:: value (bool) – Whether to sensitive for direction in processing relation pairs. Default is ‘False’

setDocLevelRelations(docLevelRelations: bool)#

Sets whether to include relations between entities from different sentences.

Parameters:: docLevelRelations (bool) – Whether to include relations between entities from different sentences.

setFilterByTokenDistance(distance: int)#

Sets number of tokens between entities. (Default: 0): Model only finds relations that have fewer than the specified number of tokens between them.

Parameters:: distance (int) – filtering criterion based on number of token between entities. (Default: 0)

setInputCols(*value)#

Sets column names of input annotations.

Parameters:: *value (List[str]) – Input columns for the annotator

setLazyAnnotator(value)#

Sets whether Annotator should be evaluated lazily in a RecursivePipeline.

Parameters:: value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline

setMaxSyntacticDistance(distance: int)#

Sets maximum syntactic distance between a pair of named entities to consider them as a relation”

Parameters:: distance (int) – Maximum syntactic distance between a pair of named entities to consider them as a relation

setOutputCol(value)#

Sets output column name of annotations.

Parameters:: value (str) – Name of output column

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

setParams()#

setRelationPairs(pairs)#

Sets list of dash-separated pairs of named entities

Parameters:: pairs (list[str] or str) – List of dash-separated pairs of named entities

setRelationPairsCaseSensitive(value: bool)#: Sets the case sensitivity of relation pairs :param value: whether relation pairs are case sensitive :type value: bool

transform(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = None) → pyspark.sql.dataframe.DataFrame#

Transforms the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

dataset (pyspark.sql.DataFrame) – input dataset
params (dict, optional) – an optional param map that overrides embedded params.

Returns:

transformed dataset

Return type:

pyspark.sql.DataFrame

write() → JavaMLWriter#: Returns an MLWriter instance for this ML instance.

sparknlp_jsl.annotator.re.relation_ner_chunk_filter#

Module Contents#

Classes#

`sparknlp_jsl.annotator.re.relation_ner_chunk_filter`#