`sparknlp_jsl.annotator.deid.light_deIdentification`#

Module Contents#

Classes#

LightDeIdentification

Light DeIdentification is a light version of DeIdentification. It replaces sensitive information

class LightDeIdentification(classname='com.johnsnowlabs.nlp.annotators.deid.LightDeIdentification', java_model=None)#

Bases: sparknlp_jsl.common.AnnotatorModelInternal

Light DeIdentification is a light version of DeIdentification. It replaces sensitive information in a text with obfuscated or masked fakers. It is designed to work with healthcare data, and it can be used to de-identify patient names, dates, and other sensitive information. It can also be used to obfuscate or mask any other type of sensitive information, such as doctor names, hospital names, and other types of sensitive information.

Additionally, it supports millions of embedded fakers and If desired, custom external fakers can be set with setCustomFakers function.

It also supports multiple languages such as English, Spanish, French, German, and Arabic. And it supports multi-mode de-Identification with setSelectiveObfuscationModes function at the same time.

Input Annotation types	Output Annotation type
`DOCUMENT, CHUNK`	`DOCUMENT`

Parameters:

mode (str) – Mode for Anonimizer [‘mask’|’obfuscate’]
dateEntities (list[str]) – List of date entities. Default: [‘DATE’, ‘DOB’, ‘DOD’]
obfuscateDate (bool) – When mode==’obfuscate’ whether to obfuscate dates or not. This param helps in consistency to make dateFormats more visible. When setting to True, make sure dateFormats param fits the needs. If the value is True and obfuscation is failed, then unnormalizedDateMode param will be activated. When setting to ‘False’, then the date will be masked to <DATE>. Default: False
unnormalizedDateMode (str) – The mode to use if the date is not formatted. Options: [mask, obfuscate, skip]. Default: obfuscate.
days (int) – A number of days to obfuscate the dates by displacement. If not provided a random integer between 1 and 60 will be used.
useShiftDays (bool) – Whether to use the random shift day when the document has this in its metadata. Default: False
dateFormats (list[str]) – List of date formats to automatically displace if parsed.
region (str) – The region to use for date parsing. This property is especially used when obfuscating dates. You can decide whether the first part of 11/11/2023 is a day or the second part is a day when obfuscating dates. Options: ‘eu’ for European Union, ‘us’ for the USA, Default: ‘eu’
obfuscateRefSource (str) – The source of obfuscation of to obfuscate the entities. For dates entities, This property is invalid. The values ar the following: custom: Takes the entities from the setCustomFakers function. faker: Takes the entities from the Faker module both : Takes the entities from the setCustomFakers function and the faker module randomly.
language (str) – The language used to select the regex file and some faker entities. The values are the following: ‘en’(English), ‘de’(German), ‘es’(Spanish), ‘fr’(French), ‘ar’(Arabic) or ‘ro’(Romanian). Default:’en’.
seed (int) – It is the seed to select the entities on obfuscate mode. With the seed, you can reply to an execution several times with the same output.
maskingPolicy (str) –

Select the masking policy:
same_length_chars: Replace the obfuscated entity with a masking sequence composed of asterisks and surrounding squared brackets, being the total length of the masking sequence of the same length as the original sequence. Example, Smith -> [***]. If the entity is less than 3 chars (like Jo, or 5), asterisks without brackets will be returned. entity_labels: Replace the values with the corresponding entity labels. fixed_length_chars: Replace the obfuscated entity with a masking sequence composed of a fixed number of asterisk.
fixedMaskLength (int) – The length of the masking sequence in case of fixed_length_chars masking policy.
sameLengthFormattedEntities (list[str]) – List of formatted entities to generate the same length outputs as original ones during obfuscation. The supported and default formatted entities are: PHONE, FAX, ID, IDNUM, BIOID, MEDICALRECORD, ZIP, VIN, SSN, DLN, LICENSE, PLATE.
genderAwareness (bool) – Whether to use gender-aware names or not during obfuscation. This param effects only names. If the value is true, it might decrease performance. Default: False
ageRanges (list[int]) – list of integer specifying limits of the age groups to preserve during obfuscation.
selectiveObfuscationModes (dict[str, list[str]]) –

The dictionary of modes to enable multi-mode deIdentification.
’obfuscate’: Replace the values with random values. ‘mask_same_length_chars’: Replace the name with the asterisks with same length minus two plus brackets on both end. ‘mask_entity_labels’: Replace the values with the entity value. ‘mask_fixed_length_chars’: Replace the name with the asterisks with fixed length. You can also invoke “setFixedMaskLength()” ‘skip’: Skip the values (intact)

The entities which have not been given in dictionary will deidentify according to :param:`mode`
customFakers (dict[str, list[str]]) – The dictionary of custom fakers to specify the obfuscation terms for the entities. You can specify the entity and the terms to be used for obfuscation.
keepYear (bool) – Whether to keep the year intact when obfuscating date entities. If True, the year will remain unchanged during the obfuscation process. If False, the year will be modified along with the month and day. Default: False.
keepMonth (bool) –
Whether to keep the month intact when obfuscating date entities.
If True, the month will remain unchanged during the obfuscation process. If False, the month will be modified along with the year and day. Default: False.

consistentAcrossNameParts: bool
Param that indicates whether consistency should be enforced across different parts of a name (e.g., first name, middle name, last name).

When set to True, the same transformation or obfuscation will be applied consistently to all parts of the same name entity, even if those parts appear separately.
For example, if “John Smith” is obfuscated as “Liam Brown”, then:
When the full name “John Smith” appears, it will be replaced with “Liam Brown”

When “John” or “Smith” appear individually, they will still be obfuscated as “Liam” and “Brown” respectively, ensuring consistency in name transformation.
Default: True

ageRanges#

dateEntities#

dateFormats#

days#

fixedMaskLength#

genderAwareness#

getter_attrs = []#

inputAnnotatorTypes#

inputCols#

keepMonth#

keepYear#

language#

lazyAnnotator#

maskingPolicy#

mode#

name = 'LightDeIdentification'#

obfuscateDate#

obfuscateRefSource#

optionalInputAnnotatorTypes = []#

outputAnnotatorType = 'document'#

outputCol#

region#

sameLengthFormattedEntities#

seed#

skipLPInputColsValidation = True#

uid = ''#

unnormalizedDateMode#

useShiftDays#

clear(param: pyspark.ml.param.Param) → None#: Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → JP#

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters:: extra (dict, optional) – Extra parameters to copy to the new instance
Returns:: Copy of this instance
Return type:: JavaParams

explainParam(param: str | Param) → str#: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:: extra (dict, optional) – extra param values
Returns:: merged param map
Return type:: dict

getInputCols()#: Gets current column names of input annotations.

getLazyAnnotator()#: Gets whether Annotator should be evaluated lazily in a RecursivePipeline.

getOrDefault(param: str) → Any#
getOrDefault(param: Param[T]) → T: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getOutputCol()#: Gets output column name of annotations.

getParam(paramName: str) → Param#: Gets a param by its name.

getParamValue(paramName)#

Gets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

hasDefault(param: str | Param[Any]) → bool#: Checks whether a param has a default value.

hasParam(paramName: str) → bool#: Tests whether this instance contains a param with a given (string) name.

inputColsValidation(value)#

isDefined(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#: Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#: Reads an ML instance from the input path, a shortcut of read().load(path).

classmethod read()#: Returns an MLReader instance for this class.

save(path: str) → None#: Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#: Sets a parameter in the embedded param map.

setAgeRanges(value: list)#

Sets list of integer specifying limits of the age groups to preserve during obfuscation.

Parameters:: value (List[int]) – List of integer specifying limits of the age groups to preserve during obfuscation.

setConsistentAcrossNameParts(value: bool)#

Sets whether to enforce consistent obfuscation across name parts, even when they appear separately. When set to True, the same transformation or obfuscation will be applied consistently to all parts of the same name entity, even if those parts appear separately.

For example, if “John Smith” is obfuscated as “Liam Brown”, then:

When the full name “John Smith” appears, it will be replaced with “Liam Brown”
When “John” or “Smith” appear individually, they will still be obfuscated as “Liam” and “Brown” respectively, ensuring consistency in name transformation.

Default: True

Parameters:: value (bool) – Whether to enforce consistent obfuscation across name parts.

setCustomFakers(value: dict)#

Sets the dictionary of custom fakers to specify the obfuscation terms for the entities. You can specify the entity and the terms to be used for obfuscation.

Example:#

>>> LightDeIdentification() \
>>>     .setObfuscateRefSource('custom') \
>>>     .setCustomFakers({'NAME': ['John', 'Doe', 'Jane'],
>>>                       'CITY': ['New York', 'Los Angeles'],
>>>                       'SCHOOL': ['Oxford', 'Harvard']})

param value:: The dictionary of custom fakers to specify the obfuscation terms for the entities.
type value:: dict[str, list[str]]

setDateEntities(entities: list)#

Sets list of date entities. Default: [‘DATE’, ‘DOB’, ‘DOD’]

Parameters:: entities (list[str]) – List of date entities.

setDateFormats(formats: list)#

Sets list of date formats to automatically displace if parsed

Parameters:: formats (list[str]) – List of date formats to automatically displace if parsed

setDays(day: int)#

Sets number of days to obfuscate by displacement the dates. If not provided a random integer between 1 and 60 will be used.

Parameters:: day (int) – Number of days to obfuscate by displacement the dates.

setFixedMaskLength(length)#: The length of the masking sequence in case of fixed_length_chars masking policy. Default: 7 :param length: The length of the masking sequence in case of fixed_length_chars masking policy. :type length: int

setForceInputTypeValidation(etfm)#

setGenderAwareness(value: bool)#

Set whether to use gender-aware names or not during obfuscation. This param effects only names. If value is true, it might decrease performance. Default: False

Parameters:: value (str) – Whether to use gender-aware names or not during obfuscation. This param effects only names. If value is true, it might decrease performance. Default: False

setInputCols(*value)#

Sets column names of input annotations.

Parameters:: *value (List[str]) – Input columns for the annotator

setKeepMonth(value: bool)#

Sets whether to keep the month intact when obfuscating date entities. If True, the month will remain unchanged during the obfuscation process. If False, the month will be modified along with the year and day. Default: False.

Parameters:: value (bool) – Whether to keep the month intact when obfuscating date entities.

setKeepYear(value: bool)#

Sets whether to keep the year intact when obfuscating date entities. If True, the year will remain unchanged during the obfuscation process. If False, the year will be modified along with the month and day. Default: False.

Parameters:: value (bool) – Whether to keep the year intact when obfuscating date entities.

setLanguage(lang: str)#

The language used to select the regex file and some faker entities. The values are the following: ‘en’(English), ‘de’(German), ‘es’(Spanish), ‘fr’(French), ‘ar’(Arabic) or ‘ro’(Romanian). Default:’en’

Parameters:: lang (str) – The language used to select the regex file and some faker entities. Default:’en’.

setLazyAnnotator(value)#

Sets whether Annotator should be evaluated lazily in a RecursivePipeline.

Parameters:: value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline

setMaskingPolicy(mask: str)#

Sets the masking policy:

same_length_chars: Replace the obfuscated entity with a masking sequence composed of asterisks and surrounding squared brackets,

being the total length of the masking sequence of the same length as the original sequence. Example, Smith -> [***]. If the entity is less than 3 chars (like Jo, or 5), asterisks without brackets will be returned.

entity_labels: Replace the values with the corresponding entity labels.
fixed_length_chars: Replace the obfuscated entity with a masking sequence composed of a fixed number of asterisks.

Parameters:: mask (str) – The masking policy

setMode(mode: str)#

Sets mode for Anonymizer [‘mask’|’obfuscate’]

Parameters:: mode (str) – Mode for Anonymizer [‘mask’|’obfuscate’]

setObfuscateDate(value: bool)#

When mode==’obfuscate’ whether to obfuscate dates or not. This param helps in consistency to make dateFormats more visible. When setting to True, make sure dateFormats param fits the needs. If the value is True and obfuscation is failed, then unnormalizedDateMode param will be activated. When setting to False, then the date will be masked to <DATE>. Default: False

Parameters:: value (bool) – When mode==’obfuscate’ whether to obfuscate dates or not. Default: False.

setObfuscateRefSource(source: str)#

The source of obfuscation to obfuscate the entities. For dates entities, This property is invalid. The values ar the following:

custom: Takes the entities from the setCustomFakers function. faker: Takes the entities from the Faker module both : Takes the entities from the setCustomFakers function and the faker module randomly.

Parameters:: source (str) – The source of obfuscation to obfuscate the entities. Default: faker.

setOutputCol(value)#

Sets output column name of annotations.

Parameters:: value (str) – Name of output column

setParamValue(paramName)#

Sets the value of a parameter.

Parameters:: paramName (str) – Name of the parameter

setParams()#

setRegion(value: str)#

With this property, you can select particular dateFormats. This property is especially used when obfuscating dates. For example: Decide whether the first part of 11/11/2023 is a day or the second part is a day when obfuscating dates. The values are following: ‘eu’ for European Union ‘us’ for the USA Default: ‘eu’

Parameters:: value (str) – The region to use select date formats. Options: ‘eu’ for European Union, ‘us’ for the USA, Default: ‘eu’

setSameLengthFormattedEntities(value: list)#

Sets list of formatted entities to generate the same length outputs as original ones during obfuscation

Parameters:

value (List[str]) –
are (The supported and default formatted entities) –

setSeed(s)#

Sets the seed to select the entities on obfuscate mode. With the seed, you can reply to an execution several times with the same output.

Parameters:: s (int) – The seed to select the entities on obfuscate mode.

setSelectiveObfuscationModes(value: dict)#

Sets the dictionary of modes to enable multi-mode deIdentification.

‘obfuscate’: Replace the values with random values.
‘mask_same_length_chars’: Replace the name with the asterisks with same length minus two plus brackets on both end.
‘mask_entity_labels’: Replace the values with the entity value.
‘mask_fixed_length_chars’: Replace the name with the asterisks with fixed length. You can also invoke “setFixedMaskLength()”
‘skip’: Skip the values (intact)

The entities which have not been given in dictionary will deidentify according to :param:`mode`

Example:#

>>> LightDeIdentification() \
>>>     .setMode('mask') \
>>>     .setSelectiveObfuscationModes({'obfuscate': ['PHONE', 'email'],
>>>                                   'mask_entity_labels': ['NAME', 'CITY'],
>>>                                   'skip': ['id']})

param value:: The dictionary of modes to enable multi-mode deIdentification.
type value:: dict[str, list[str]]

setUnnormalizedDateMode(mode: str)#

Sets the mode to use if the date is not formatted. Options: [mask, obfuscate, skip]. Default: obfuscate.

Parameters:: mode (str) – The mode to use if the date is not formatted.

setUseShiftDays(value: bool)#

Sets if you want to use the random shift day when the document has this in its metadata. Default: False

Parameters:: value (bool) – Whether to use the random shift day when the document has this in its metadata. Default: False

transform(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = None) → pyspark.sql.dataframe.DataFrame#

Transforms the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

dataset (pyspark.sql.DataFrame) – input dataset
params (dict, optional) – an optional param map that overrides embedded params.

Returns:

transformed dataset

Return type:

pyspark.sql.DataFrame

write() → JavaMLWriter#: Returns an MLWriter instance for this ML instance.

sparknlp_jsl.annotator.deid.light_deIdentification#

Module Contents#

Classes#

Example:#

Example:#

`sparknlp_jsl.annotator.deid.light_deIdentification`#