trait DeIdentificationParams extends BaseDeidParams with MaskingParams with HasFeatures
A trait that contains all the params that are common between DeIdentificationModel and DeIdentification annotators.
- Grouped
- Alphabetic
- By Inheritance
- DeIdentificationParams
- MaskingParams
- BaseDeidParams
- HasFeatures
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Abstract Value Members
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
val
additionalDateFormats: StringArrayParam
Additional date formats to be considered during date obfuscation.
Additional date formats to be considered during date obfuscation. This allows users to specify custom date formats in addition to the default dateFormats.
- Definition Classes
- BaseDeidParams
-
val
ageGroups: StructFeature[Map[String, Array[Int]]]
A map of age groups to obfuscate ages.
A map of age groups to obfuscate ages. For this parameter to be active, the obfuscateByAgeGroups parameter must be true. If the given ageGroups do not fully contain the ages, the ages continue to be obfuscated according to the ageRanges. The map should contain the age group name as the key and an array of two integers as the value. The first integer is the lower bound of the age group, and the second integer is the upper bound of the age group. Default age groups are as follows in the English language:
Map( "baby" -> Array(0, 1), "toddler" -> Array(1, 4), "child" -> Array(4, 13), "teenager" -> Array(13, 20), "adult" -> Array(20, 65), "senior" -> Array(65, 200) )
-
val
ageRanges: IntArrayParam
List of integers specifying limits of the age groups to preserve during obfuscation
List of integers specifying limits of the age groups to preserve during obfuscation
- Definition Classes
- BaseDeidParams
-
val
ageRangesByHipaa: BooleanParam
A Boolean variable indicating whether to obfuscate ages based on HIPAA (Health Insurance Portability and Accountability Act) Privacy Rule.
A Boolean variable indicating whether to obfuscate ages based on HIPAA (Health Insurance Portability and Accountability Act) Privacy Rule.
The HIPAA Privacy Rule mandates that ages from patients older than 90 years must be obfuscated, while age for patients 90 years or younger can remain unchanged.
When
true
, age entities larger than 90 will be obfuscated as per HIPAA Privacy Rule, the others will remain unchanged. Whenfalse
, ageRanges parameter is valid.- Definition Classes
- BaseDeidParams
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
val
blackList: StringArrayParam
List of entities that will be ignored in the regex file.
List of entities that will be ignored in the regex file. The rest will be processed. The default values are "IBAN","ZIP","NPI","DLN","PASSPORT","C_CARD","DEA","SSN", "IP", "DEA".
-
val
blackListEntities: StringArrayParam
List of entities coming from NER or regex rules that will be ignored for masking or obfuscation.
List of entities coming from NER or regex rules that will be ignored for masking or obfuscation. The rest entities will be processed. Defaults to an empty array.
-
val
chunkMatching: MapFeature[String, Double]
Performs entity chunk matching across rows or within groups in a DataFrame.
Performs entity chunk matching across rows or within groups in a DataFrame. Useful in de-identification pipelines where certain entity labels like "NAME" or "DATE" may be missing in some rows and need to be filled from other rows in the same group.
- Note
When applying the method across multiple rows, the usage of
groupByCol
parameter is required.
-
final
def
clear(param: Param[_]): DeIdentificationParams.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
lazy val
combinedDateFormats: Array[String]
- Attributes
- protected
- Definition Classes
- BaseDeidParams
-
val
consistentAcrossNameParts: BooleanParam
Param that indicates whether consistency should be enforced across different parts of a name (e.g., first name, middle name, last name).
Param that indicates whether consistency should be enforced across different parts of a name (e.g., first name, middle name, last name). When set to
true
, the same transformation or obfuscation will be applied consistently to all parts of the same name entity, even if those parts appear separately.For example, if "John Smith" is obfuscated as "Liam Brown", then:
- When the full name "John Smith" appears, it will be replaced with "Liam Brown"
- When "John" or "Smith" appear individually, they will still be obfuscated as "Liam" and "Brown" respectively, ensuring consistency in name transformation.
Default: true
- Definition Classes
- BaseDeidParams
-
val
consistentObfuscation: BooleanParam
Whether to replace very similar entities in a document with the same randomized term (default: true) The similarity is based on the Levenshtein Distance between the words.
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
val
countryObfuscation: BooleanParam
Whether to obfuscate country entities or not.
Whether to obfuscate country entities or not. If true, country entities will be obfuscated using the Faker module. If false, country entities will be skipped during obfuscation. Default: false
- Definition Classes
- BaseDeidParams
-
val
dateEntities: StringArrayParam
List of date entities.
List of date entities. Default: Array("DATE", "DOB", "DOD", "EFFDATE", "FISCAL_YEAR")
- Definition Classes
- BaseDeidParams
-
val
dateFormats: StringArrayParam
Format of dates to displace
Format of dates to displace
- Definition Classes
- BaseDeidParams
-
val
dateTag: Param[String]
Tag representing what are the NER entity (default: DATE)
-
val
dateToYear: BooleanParam
true if dates must be converted to years, false otherwise
-
val
days: IntParam
Number of days to obfuscate the dates by displacement.
Number of days to obfuscate the dates by displacement. If not provided a random integer between 1 and 60 will be used
- Definition Classes
- BaseDeidParams
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
fakerLengthOffset: IntParam
It specifies how much length deviation is accepted in obfuscation, with keepTextSizeForObfuscation enabled.
It specifies how much length deviation is accepted in obfuscation, with keepTextSizeForObfuscation enabled. Value must be greater than 0. Default is 3.
- Definition Classes
- BaseDeidParams
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
val
fixedMaskLength: IntParam
Select the fixed mask length: this is the length of the masking sequence that will be used when the 'fixed_length_chars' masking policy is selected.
Select the fixed mask length: this is the length of the masking sequence that will be used when the 'fixed_length_chars' masking policy is selected.
- Definition Classes
- MaskingParams
-
val
genderAwareness: BooleanParam
Whether to use gender-aware names or not during obfuscation.
Whether to use gender-aware names or not during obfuscation. This param effects only names. If value is true, it might decrease performance. Default: False
- Definition Classes
- BaseDeidParams
-
val
geoConsistency: BooleanParam
Whether to enforce consistent obfuscation across geographical entities: state, city, street, zip and phone.
Whether to enforce consistent obfuscation across geographical entities: state, city, street, zip and phone.
## Functionality Overview This parameter enables intelligent geographical entity obfuscation that maintains realistic relationships between different geographic components. When enabled, the system ensures that obfuscated addresses form coherent, valid combinations rather than random replacements.
## Supported Entity Types The following geographical entities are processed with priority order: - **state** (Priority: 0) - US state names - **city** (Priority: 1) - City names - **zip** (Priority: 2) - Zip codes - **street** (Priority: 3) - Street addresses - **phone** (Priority: 4) - Phone numbers
## Language Requirement **IMPORTANT**: Geographic consistency is only applied when: -
geoConsistency
parameter is set totrue
AND -language
parameter is set toen
For non-English configurations, this feature is automatically disabled regardless of the parameter setting.
## Consistency Algorithm When geographical entities comes from the chunk columns:
1. **Entity Grouping**: All geographic entities are identified and grouped by type 2. **Fake Address Selection**: A consistent set of fake US addresses is selected using hash-based deterministic selection to ensure reproducibility 3. **Priority-Based Mapping**: Entities are mapped to fake addresses following the priority order (state → city → zip → street → phone) 4. **Consistent Replacement**: All entities of the same type within a document use the same fake address pool, maintaining geographical coherence
## Parameter Interactions **IMPORTANT**: Enabling this parameter automatically disables: -
keepTextSizeForObfuscation
- Text size preservation is not maintained -consistentObfuscation
- Standard consistency rules are overridden - file-based fakersThis is necessary because geographic consistency requires specific fake address selection that may not preserve original text lengths or follow standard obfuscation patterns.
default: false
- Definition Classes
- BaseDeidParams
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getAdditionalDateFormats: Array[String]
Gets the value of additionalDateFormats
Gets the value of additionalDateFormats
- Definition Classes
- BaseDeidParams
-
def
getAgeRanges: Array[Int]
Gets ageRanges param.
Gets ageRanges param.
- Definition Classes
- BaseDeidParams
-
def
getAgeRangesByHipaa: Boolean
Gets the value of ageRangesByHipaa.
Gets the value of ageRangesByHipaa.
- Definition Classes
- BaseDeidParams
-
def
getBlackListEntities: Array[String]
Gets blackListEntities param
- def getChunkMatching: Map[String, Double]
- def getChunkMatchingAsStr: String
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getConsistentAcrossNameParts: Boolean
Gets the value of consistentAcrossNameParts.
Gets the value of consistentAcrossNameParts.
- Definition Classes
- BaseDeidParams
- def getConsistentObfuscation: Boolean
-
def
getCountryObfuscation: Boolean
Gets the value of countryObfuscation.
Gets the value of countryObfuscation.
- Definition Classes
- BaseDeidParams
-
def
getDateEntities: Array[String]
Gets dateEntities param.
Gets dateEntities param.
- Definition Classes
- BaseDeidParams
-
def
getDateFormats: Array[String]
Gets the value of dateFormats
Gets the value of dateFormats
- Definition Classes
- BaseDeidParams
- def getDateTag: String
- def getDateToYear: Boolean
-
def
getDays: Int
Gets days param
Gets days param
- Definition Classes
- BaseDeidParams
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getEntityBasedObfuscationRefSource(entityClass: String): String
- Attributes
- protected
- Definition Classes
- BaseDeidParams
-
def
getFakerLengthOffset: Int
Gets fakerLengthOffset param
Gets fakerLengthOffset param
- Definition Classes
- BaseDeidParams
-
def
getFixedMaskLength: Int
Gets fixedMaskLength param.
Gets fixedMaskLength param.
- Definition Classes
- MaskingParams
-
def
getGenderAwareness: Boolean
Gets genderAwareness param.
Gets genderAwareness param.
- Definition Classes
- BaseDeidParams
-
def
getGeoConsistency: Boolean
Gets the value of geoConsistency.
Gets the value of geoConsistency.
- Definition Classes
- BaseDeidParams
-
def
getGroupByCol: String
Gets groupByCol param
- def getIgnoreRegex: Boolean
-
def
getKeepMonth: Boolean
Gets keepMonth param
Gets keepMonth param
- Definition Classes
- BaseDeidParams
-
def
getKeepTextSizeForObfuscation: Boolean
Gets keepTextSizeForObfuscation param
Gets keepTextSizeForObfuscation param
- Definition Classes
- BaseDeidParams
-
def
getKeepYear: Boolean
Gets keepYear param
Gets keepYear param
- Definition Classes
- BaseDeidParams
-
def
getLanguage: String
Gets language param.
Gets language param.
- Definition Classes
- BaseDeidParams
- def getMappingsColumn: String
-
def
getMaskStatus(entityClass: String): String
- Attributes
- protected
- Definition Classes
- MaskingParams
-
def
getMaskingPolicy: String
Gets maskingPolicy param.
Gets maskingPolicy param.
- Definition Classes
- MaskingParams
-
def
getMetadataMaskingPolicy: String
Gets metadataMaskingPolicy param
- def getMinYear: Int
-
def
getMode: String
Gets mode param.
Gets mode param.
- Definition Classes
- BaseDeidParams
-
def
getObfuscateByAgeGroups: Boolean
Gets obfuscateByAgeGroups param
-
def
getObfuscateDate: Boolean
Gets obfuscateDate param
Gets obfuscateDate param
- Definition Classes
- BaseDeidParams
-
def
getObfuscateRefSource: String
Gets obfuscateRefSource param.
Gets obfuscateRefSource param.
- Definition Classes
- BaseDeidParams
- def getObfuscationStrategyOnException: String
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
- def getRegexOverride: Boolean
-
def
getRegion: String
Gets region param.
Gets region param.
- Definition Classes
- BaseDeidParams
- def getReturnEntityMappings: Boolean
- def getSameEntityThreshold: Double
-
def
getSameLengthFormattedEntities(): Array[String]
- Definition Classes
- BaseDeidParams
-
def
getSeed(): Int
- Definition Classes
- BaseDeidParams
-
def
getSelectiveObfuscateRefSource: Map[String, String]
Gets selectiveObfuscateRefSource param.
Gets selectiveObfuscateRefSource param.
- Definition Classes
- BaseDeidParams
-
def
getSelectiveObfuscateRefSourceAsStr: String
- Definition Classes
- BaseDeidParams
-
def
getStaticObfuscationPairs: Option[Array[StaticObfuscationEntity]]
- Definition Classes
- BaseDeidParams
-
def
getUnnormalizedDateMode: String
Gets unnormalizedDateMode param.
Gets unnormalizedDateMode param.
- Definition Classes
- BaseDeidParams
-
def
getUseShiftDays: Boolean
Getter method of useShiftDays
Getter method of useShiftDays
- Definition Classes
- DeIdentificationParams → BaseDeidParams
-
def
getValidAgeRanges: Array[Int]
Gets valid ageRanges whether ageRangesByHipaa is true or not.
Gets valid ageRanges whether ageRangesByHipaa is true or not.
- Attributes
- protected
- Definition Classes
- BaseDeidParams
- def getZipCodeTag: String
-
val
groupByCol: Param[String]
The column name used to group the dataset.
The column name used to group the dataset. This parameter is used in conjunction with
consistentObfuscation
to ensure consistent obfuscation within each group. WhengroupByCol
is set, the dataset is partitioned into groups based on the values of the specified column.Default:
""
(empty string, meaning no grouping)- The column name must be a valid string in the input dataset.
- The column must be of
StringType
.
- Note
This functionality can change order of the dataset, so it is recommended to use it with caution.
,This functionality cannot be supported by LightPipeline.
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
val
ignoreRegex: BooleanParam
Select if you want to use regex file loaded in the model.
Select if you want to use regex file loaded in the model. If true the default regex file will be not used The default value is false.
-
def
isArabic: Boolean
- Attributes
- protected
- Definition Classes
- MaskingParams
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
val
isRandomDateDisplacement: BooleanParam
Use a random displacement days in dates entities,that random number is based on the DeIdentificationParams.seed If true use random displacement days in dates entities,if false use the DeIdentificationParams.days The default value is false.
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
val
keepMonth: BooleanParam
Whether to keep the month intact when obfuscating date entities.
Whether to keep the month intact when obfuscating date entities. If true, the month will remain unchanged during the obfuscation process. If false, the month will be modified along with the year and day. Default: false.
- Definition Classes
- BaseDeidParams
-
val
keepTextSizeForObfuscation: BooleanParam
It specifies whether the output should maintain the same character length as the input text.
It specifies whether the output should maintain the same character length as the input text. the output text will remain the same if same length is available, else length might vary.
- Definition Classes
- BaseDeidParams
-
val
keepYear: BooleanParam
Whether to keep the year intact when obfuscating date entities.
Whether to keep the year intact when obfuscating date entities. If true, the year will remain unchanged during the obfuscation process. If false, the year will be modified along with the month and day. Default: false.
- Definition Classes
- BaseDeidParams
-
val
language: Param[String]
The language used to select the regex file and some faker entities.
The language used to select the regex file and some faker entities. 'en'(English),'de'(German), 'es'(Spanish), 'fr'(French), 'ar'(Arabic) or 'ro'(Romanian) Default:'en'
- Definition Classes
- BaseDeidParams
-
val
mappingsColumn: Param[String]
This is the mapping column that will return the Annotations chunks with the fake entities
-
def
maskEntity(wordToReplace: String, entityClass: String): String
- Attributes
- protected
- Definition Classes
- MaskingParams
-
def
maskEntity(annotation: Annotation, entityClass: String): String
- Attributes
- protected
- Definition Classes
- MaskingParams
-
def
maskEntityWithPolicy(wordToReplace: String, maskingPolicy: String, entityClass: String): String
- Attributes
- protected
- Definition Classes
- MaskingParams
-
def
maskEntityWithPolicy(annotation: Annotation, maskingPolicy: String, entityClass: String): String
- Attributes
- protected
- Definition Classes
- MaskingParams
-
val
maskingPolicy: Param[String]
Select the masking policy:
Select the masking policy:
- 'entity_labels': Replace the values with the entity value.
- 'same_length_chars': Replace the name with the asterix with same length minus two plus brackets on both end.If the entity is less than 3 chars (like Jo, or 5), we can just use asterix without brackets.
- 'fixed_length_chars': Replace the obfuscated entity with a masking sequence composed of a fixed number of asterisk.
- 'entity_labels_without_brackets': Replace the values with the entity value without brackets.
- 'same_length_chars_without_brackets': Replace the name with the asterix with same length without brackets.
- Default: 'entity_labels'
- Definition Classes
- MaskingParams
-
val
metadataMaskingPolicy: Param[String]
If specified, the metadata includes the masked form of the document.
If specified, the metadata includes the masked form of the document. Select the following masking policy if you want to return mask form in the metadata:
- 'entity_labels': Replace the values with the entity value.
- 'same_length_chars': Replace the name with the asterix with same length minus two plus brackets on both end.If the entity is less than 3 chars (like Jo, or 5), we can just use asterix without brackets.
- 'fixed_length_chars': Replace the obfuscated entity with a masking sequence composed of a fixed number of asterisk.
- 'entity_labels_without_brackets': Replace the values with the entity value without brackets.
- 'same_length_chars_without_brackets': Replace the name with the asterix with same length without brackets.
- Default: ""
-
val
minYear: IntParam
Minimum year to use when converting date to year
-
val
mode: Param[String]
Mode for Anonymizer ['mask' or 'obfuscate'].
Mode for Anonymizer ['mask' or 'obfuscate']. Default: 'mask'
- Mask mode: The entities will be replaced by their entity types.
- Obfuscate mode: The entity is replaced by an obfuscator's term.
- Definition Classes
- BaseDeidParams
Given the following text: "David Hale visited EEUU a couple of years ago"
- Mask mode: "
<
PERSON>
visited<
COUNTRY>
a couple of years ago" - Obfuscate mode: "Bryan Johnson visited Japan a couple of years ago"
- Mask mode: "
Example: -
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
val
obfuscateByAgeGroups: BooleanParam
Whether to obfuscate ages based on age groups.
-
val
obfuscateDate: BooleanParam
When mode=="obfuscate" whether to obfuscate dates or not.
When mode=="obfuscate" whether to obfuscate dates or not. This param helps in consistency to make dateFormats more visible. When setting to
true
, make sure dateFormats param fits the needs. If the value is true and obfuscation is failed, then unnormalizedDateMode will be activated. When setting to 'false', then the date will be masked to <DATE>. Default: false- Definition Classes
- BaseDeidParams
-
val
obfuscateRefSource: Param[String]
The source of obfuscation to obfuscate the entities.
The source of obfuscation to obfuscate the entities. The values ar the following: 'file': Takes the entities from the obfuscatorRefFile 'faker': Takes the entities from the Faker module 'both': Takes the entities from the obfuscatorRefFile and the faker module randomly.
- Definition Classes
- BaseDeidParams
-
val
obfuscationStrategyOnException: Param[String]
The obfuscation strategy to be applied when an exception occurs.
The obfuscation strategy to be applied when an exception occurs.
The obfuscation strategy determines how obfuscation is handled in case of an exception. Four possible values are supported:
- "mask": The original chunk is replaced with a masking pattern.
- "default": The original chunk is replaced with a default faker.
- "skip": The original chunk is not replaced with any faker.
- "exception": Throws the exception.
The default obfuscation strategy is "default".
-
val
outputAsDocument: BooleanParam
Whether to return all sentences joined into a single document
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
lazy val
randomDateFormat: String
- Attributes
- protected
- Definition Classes
- BaseDeidParams
-
val
regexOverride: BooleanParam
If the value is true, prioritize the regex entities; if the value is false, prioritize the ner.
If the value is true, prioritize the regex entities; if the value is false, prioritize the ner. The default value is false. If DeIdentification.combineRegexPatterns is true, this value will be invalid.
-
val
region: Param[String]
With this property, you can select particular dateFormats.
With this property, you can select particular dateFormats. This property is especially used when obfuscating dates. You can decide whether the first part of 11/11/2023 is a day or the second part is a day when obfuscating dates.
- The values are following:
- 'eu' for European Union
- 'us' for USA
- Definition Classes
- BaseDeidParams
-
val
returnEntityMappings: BooleanParam
With this property, you can select if you want to return mapping column.
-
val
sameEntityThreshold: DoubleParam
Similarity threshold [0.0-1.0] to consider two appearances of an entity as
the same
(default: 0.9) For date entities this method doesn't apply. -
val
sameLengthFormattedEntities: StringArrayParam
List of formatted entities to generate the same length outputs as original ones during obfuscation.
List of formatted entities to generate the same length outputs as original ones during obfuscation. The supported and default formatted entities are: "phone", "fax", "contact," "id", "idnum", "bioid", "medicalrecord", "zip", "vin", "ssn", "dln", "plate", "license", "IRS", "CFN", "account".
- Definition Classes
- BaseDeidParams
-
val
seed: IntParam
It is the seed to select the entities on obfuscate mode.
It is the seed to select the entities on obfuscate mode. With the seed, you can reply to an execution several times with the same output.
- Definition Classes
- BaseDeidParams
-
val
selectiveObfuscateRefSource: MapFeature[String, String]
A map of entity names to their obfuscation modes.
A map of entity names to their obfuscation modes. This is used to selectively apply different obfuscation methods to specific entities. The keys are entity names and the values are the obfuscation sources. If an entity is not specified in this map, the obfuscateRefSource param is used to determine the obfuscation source.
- Definition Classes
- BaseDeidParams
val selectiveSources = Map( "PHONE" -> "file", "EMAIL" -> "faker", "NAME" -> "faker", "ADDRESS" -> "both" )
Example: -
def
set[T](feature: StructFeature[T], value: T): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): DeIdentificationParams.this.type
- Definition Classes
- Params
-
def
setAdditionalDateFormats(formats: Array[String]): DeIdentificationParams.this.type
Sets additionalDateFormats param
Sets additionalDateFormats param
- Definition Classes
- BaseDeidParams
-
def
setAgeGroups(value: Map[String, Array[Int]]): DeIdentificationParams.this.type
Sets the age groups to obfuscate ages.
Sets the age groups to obfuscate ages. For this parameter to be active, the obfuscateByAgeGroups parameter must be true. If the given ageGroups do not fully contain the ages, the ages continue to be obfuscated according to the ageRanges. The map should contain the age group name as the key and an array of two integers as the value. The first integer is the lower bound of the age group, and the second integer is the upper bound of the age group. Default age groups are as follows in the English language:
Map( "baby" -> Array(0, 1), "toddler" -> Array(1, 3), "child" -> Array(3, 12), "teenager" -> Array(12, 20), "adult" -> Array(20, 65), "senior" -> Array(65, 200) )
- Exceptions thrown
IllegalArgumentException
if the value is empty, contains negative values, or is not a pair of integers
- def setAgeGroups(value: HashMap[String, ArrayList[Int]]): DeIdentificationParams.this.type
-
def
setAgeRanges(mode: Array[Int]): DeIdentificationParams.this.type
List of integers specifying limits of the age groups to preserve during obfuscation
List of integers specifying limits of the age groups to preserve during obfuscation
- Definition Classes
- BaseDeidParams
-
def
setAgeRangesByHipaa(value: Boolean): DeIdentificationParams.this.type
Sets whether to obfuscate ages based on HIPAA (Health Insurance Portability and Accountability Act) Privacy Rule.
Sets whether to obfuscate ages based on HIPAA (Health Insurance Portability and Accountability Act) Privacy Rule.
The HIPAA Privacy Rule mandates that ages from patients older than 90 years must be obfuscated, while age for patients 90 years or younger can remain unchanged.
- value
If true, age entities larger than 90 will be obfuscated as per HIPAA Privacy Rule, the others will remain unchanged. If false, ageRanges parameter is valid. Default: false.
- Definition Classes
- BaseDeidParams
-
def
setBlackList(list: Array[String]): DeIdentificationParams.this.type
List of entities that will be ignored to in the regex file.
List of entities that will be ignored to in the regex file. The rest will be processed. The default values are "IBAN","ZIP","NPI","DLN","PASSPORT","C_CARD","DEA","SSN", "IP", "DEA".
-
def
setBlackListEntities(value: Array[String]): DeIdentificationParams.this.type
Sets the list of entities coming from NER or regex rules that will be ignored for masking or obfuscation.
Sets the list of entities coming from NER or regex rules that will be ignored for masking or obfuscation. The rest entities will be processed. Defaults to an empty array.
- def setChunkMatching(categories: HashMap[String, Double]): DeIdentificationParams.this.type
-
def
setChunkMatching(value: Map[String, Double]): DeIdentificationParams.this.type
Performs entity chunk matching across rows or within groups in a DataFrame.
Performs entity chunk matching across rows or within groups in a DataFrame. Useful in de-identification pipelines where certain entity labels like "NAME" or "DATE" may be missing in some rows and need to be filled from other rows in the same group.
Notes:
- When applying the method across multiple rows, the usage of
groupByCol
parameter is required.
- When applying the method across multiple rows, the usage of
-
def
setConsistentAcrossNameParts(value: Boolean): DeIdentificationParams.this.type
Sets the value of consistentAcrossNameParts.
Sets the value of consistentAcrossNameParts.
- value
Boolean flag to enforce consistency across name parts
- returns
this instance
- Definition Classes
- BaseDeidParams
-
def
setConsistentObfuscation(s: Boolean): DeIdentificationParams.this.type
Whether to replace very similar entities in a document with the same randomized term (default: true) The similarity is based on the Levenshtein Distance between the words.
-
def
setCountryObfuscation(value: Boolean): DeIdentificationParams.this.type
Sets whether to obfuscate country entities or not.
Sets whether to obfuscate country entities or not. If true, country entities will be obfuscated using the Faker module. If false, country entities will be skipped during obfuscation. Default: false
- Definition Classes
- BaseDeidParams
-
def
setDateEntities(value: Array[String]): DeIdentificationParams.this.type
Sets the value of dateEntities.
Sets the value of dateEntities. Default: Array("DATE", "DOB", "DOD", "EFFDATE", "FISCAL_YEAR")
- Definition Classes
- BaseDeidParams
-
def
setDateFormats(s: Array[String]): DeIdentificationParams.this.type
Format of dates to displace
Format of dates to displace
- Definition Classes
- BaseDeidParams
-
def
setDateTag(s: String): DeIdentificationParams.this.type
Tag representing what are the NER entity (default: DATE)
-
def
setDateToYear(s: Boolean): DeIdentificationParams.this.type
true if dates must be converted to years, false otherwise
-
def
setDays(k: Int): DeIdentificationParams.this.type
Number of days to obfuscate the dates by displacement.
Number of days to obfuscate the dates by displacement. If not provided a random integer between 1 and 60 will be used
- Definition Classes
- BaseDeidParams
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): DeIdentificationParams.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): DeIdentificationParams.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
-
def
setFakerLengthOffset(value: Int): DeIdentificationParams.this.type
Sets fakerLengthOffset param
Sets fakerLengthOffset param
- Definition Classes
- BaseDeidParams
-
def
setFixedMaskLength(value: Int): DeIdentificationParams.this.type
Sets the value of fixedMaskLength.
Sets the value of fixedMaskLength. This is the length of the masking sequence that will be used when the 'fixed_length_chars' masking policy is selected.
- Definition Classes
- MaskingParams
-
def
setGenderAwareness(value: Boolean): DeIdentificationParams.this.type
Whether to use gender-aware names or not during obfuscation.
Whether to use gender-aware names or not during obfuscation. This param effects only names. If value is true, it might decrease performance. Default: False
- Definition Classes
- BaseDeidParams
-
def
setGeoConsistency(value: Boolean): DeIdentificationParams.this.type
Sets the value of geoConsistency.
Sets the value of geoConsistency. When set to
true
, it enables consistent obfuscation across geographical entities such as state, city, street, zip, and phone.- Definition Classes
- BaseDeidParams
-
def
setGroupByCol(value: String): DeIdentificationParams.this.type
Sets groupByCol param to group the dataset.
Sets groupByCol param to group the dataset. This parameter is used in conjunction with
consistentObfuscation
to ensure consistent obfuscation within each group.- Note
This functionality can change order of the dataset, so it is recommended to use it with caution.
,This functionality cannot be supported by LightPipeline.
-
def
setIgnoreRegex(s: Boolean): DeIdentificationParams.this.type
Select if you want to use regex file loaded in the model.
Select if you want to use regex file loaded in the model. If true the default regex file will be not used The default value is false.
-
def
setIsRandomDateDisplacement(s: Boolean): DeIdentificationParams.this.type
Use a random displacement days in dates entities,that random number is based on the DeIdentificationParams.seed If true use random displacement days in dates entities, if false use the DeIdentificationParams.days The default value is false.
-
def
setKeepMonth(value: Boolean): DeIdentificationParams.this.type
Sets whether to keep the month intact when obfuscating date entities.
Sets whether to keep the month intact when obfuscating date entities. If true, the month will remain unchanged during the obfuscation process. If false, the month will be modified along with the year and day. Default: false.
- Definition Classes
- BaseDeidParams
-
def
setKeepTextSizeForObfuscation(value: Boolean): DeIdentificationParams.this.type
Sets keepTextSizeForObfuscation param
Sets keepTextSizeForObfuscation param
- Definition Classes
- BaseDeidParams
-
def
setKeepYear(value: Boolean): DeIdentificationParams.this.type
Sets whether to keep the year intact when obfuscating date entities.
Sets whether to keep the year intact when obfuscating date entities. If true, the year will remain unchanged during the obfuscation process. If false, the year will be modified along with the month and day. Default: false.
- Definition Classes
- BaseDeidParams
-
def
setLanguage(s: String): DeIdentificationParams.this.type
The language used to select the regex file and some faker entities.
The language used to select the regex file and some faker entities. 'en'(English),'de'(German), 'es'(Spanish), 'fr'(French), 'ar'(Arabic) or 'ro'(Romanian). Default:'en'
- Definition Classes
- BaseDeidParams
-
def
setMappingsColumn(s: String): DeIdentificationParams.this.type
This is the mapping column that will return the Annotations chunks with the fake entities
-
def
setMaskingPolicy(value: String): DeIdentificationParams.this.type
Select the masking policy:
Select the masking policy:
- 'entity_labels': Replace the values with the entity value.
- 'same_length_chars': Replace the name with the asterix with same length minus two plus brackets on both end.If the entity is less than 3 chars (like Jo, or 5), we can just use asterix without brackets.
- 'fixed_length_chars': Replace the obfuscated entity with a masking sequence composed of a fixed number of asterisk.
- 'entity_labels_without_brackets': Replace the values with the entity value without brackets.
- 'same_length_chars_without_brackets': Replace the name with the asterix with same length without brackets.
- Default: 'entity_labels'
- Definition Classes
- MaskingParams
-
def
setMetadataMaskingPolicy(value: String): DeIdentificationParams.this.type
If specified, the metadata includes the masked form of the document.
If specified, the metadata includes the masked form of the document. Select the following masking policy if you want to return mask form in the metadata:
- 'entity_labels': Replace the values with the entity value.
- 'same_length_chars': Replace the name with the asterix with same length minus two plus brackets on both end.If the entity is less than 3 chars (like Jo, or 5), we can just use asterix without brackets.
- 'fixed_length_chars': Replace the obfuscated entity with a masking sequence composed of a fixed number of asterisk.
- 'entity_labels_without_brackets': Replace the values with the entity value without brackets.
- 'same_length_chars_without_brackets': Replace the name with the asterix with same length without brackets.
- Default: ""
-
def
setMinYear(s: Int): DeIdentificationParams.this.type
Minimum year to use when converting date to year
-
def
setMode(m: String): DeIdentificationParams.this.type
Mode for Anonymizer ['mask'|'obfuscate'].
Mode for Anonymizer ['mask'|'obfuscate']. Default: 'mask'
- Mask mode: The entities will be replaced by their entity types.
- Obfuscate mode: The entity is replaced by an obfuscator's term.
- Definition Classes
- BaseDeidParams
Given the following text: "David Hale visited EEUU a couple of years ago"
- Mask mode: "
<
PERSON>
visited<
COUNTRY>
a couple of years ago" - Obfuscate mode: "Bryan Johnson visited Japan a couple of years ago"
- Mask mode: "
Example: -
def
setObfuscateByAgeGroups(value: Boolean): DeIdentificationParams.this.type
Sets whether to obfuscate ages based on age groups.
-
def
setObfuscateDate(s: Boolean): DeIdentificationParams.this.type
When mode=="obfuscate" whether to obfuscate dates or not.
When mode=="obfuscate" whether to obfuscate dates or not. This param helps in consistency to make dateFormats more visible. When setting to
true
, make sure dateFormats param fits the needs. If the value is true and obfuscation is failed, then unnormalizedDateMode will be activated. When setting to 'false' then the date will be masked to <DATE> . Default: false- Definition Classes
- BaseDeidParams
-
def
setObfuscateRefSource(s: String): DeIdentificationParams.this.type
The source of obfuscation to obfuscate the entities.
The source of obfuscation to obfuscate the entities. The values are the following: 'file': Takes the fakes from the obfuscatorRefFile 'faker': Takes the fakes from the Faker module 'both': Takes the fakes from the obfuscatorRefFile and the faker module randomly.
- Definition Classes
- BaseDeidParams
-
def
setObfuscationStrategyOnException(value: String): DeIdentificationParams.this.type
Sets the obfuscation strategy to be applied when an exception occurs.
Sets the obfuscation strategy to be applied when an exception occurs.
The obfuscation strategy determines how obfuscation is handled in case of an exception. Four possible values are supported:
- "mask": The original chunk is replaced with a masking pattern.
- "default": The original chunk is replaced with a default faker.
- "skip": The original chunk is not replaced with any faker.
- "exception": Throws the exception.
The default obfuscation strategy is "default".
-
def
setOutputAsDocument(mode: Boolean): DeIdentificationParams.this.type
Whether to return all sentences joined into a single document
-
def
setRegexOverride(s: Boolean): DeIdentificationParams.this.type
If the value is true, prioritize the regex entities; if the value is false, prioritize the ner.
If the value is true, prioritize the regex entities; if the value is false, prioritize the ner. The default value is false. If DeIdentification.combineRegexPatterns is true, this value will be invalid.
-
def
setRegion(s: String): DeIdentificationParams.this.type
With this property, you can select particular dateFormats.
With this property, you can select particular dateFormats. This property is especially used when obfuscating dates. You can decide whether the first part of 11/11/2023 is a day or the second part is a day when obfuscating dates. The values are following:
- 'eu' for European Union
- 'us' for USA
- Definition Classes
- BaseDeidParams
-
def
setReturnEntityMappings(s: Boolean): DeIdentificationParams.this.type
With this property, you can select if you want to return mapping column.
-
def
setSameEntityThreshold(s: Double): DeIdentificationParams.this.type
Similarity threshold [0.0-1.0] to consider two appearances of an entity as
the same
(default: 0.9) For date entities this method doesn't apply. -
def
setSameLengthFormattedEntities(entities: Array[String]): DeIdentificationParams.this.type
List of formatted entities to generate the same length outputs as original ones during obfuscation.
List of formatted entities to generate the same length outputs as original ones during obfuscation. The supported and default formatted entities are: PHONE, FAX, CONTACT, ID, IDNUM, BIOID, MEDICALRECORD, ZIP, VIN, SSN, DLN, LICENSE, PLATE, IRS, CFN, ACCOUNT.
- Definition Classes
- BaseDeidParams
-
def
setSeed(s: Int): DeIdentificationParams.this.type
It is the seed to select the entities on obfuscate mode.
It is the seed to select the entities on obfuscate mode. With the seed, you can reply to an execution several times with the same output.
- Definition Classes
- BaseDeidParams
-
def
setSelectiveObfuscateRefSource(value: HashMap[String, String]): DeIdentificationParams.this.type
- Definition Classes
- BaseDeidParams
-
def
setSelectiveObfuscateRefSource(value: Map[String, String]): DeIdentificationParams.this.type
Sets the value of selectiveObfuscateRefSource.
Sets the value of selectiveObfuscateRefSource. This is used to selectively apply different obfuscation methods to specific entities. The keys are entity names and the values are the obfuscation sources. If an entity is not specified in this map, the obfuscateRefSource param is used to determine the obfuscation method. The values can be: - 'file': Takes the fakes from the file. - 'faker': Takes the fakes from the embedded faker module. - 'both': Takes the fakes from the file and the faker module.
- Definition Classes
- BaseDeidParams
val modes = Map( "PHONE" -> "file", "EMAIL" -> "faker", "NAME" -> "faker", "ADDRESS" -> "both" )
Example: -
def
setStaticObfuscationPairs(pairs: ArrayList[ArrayList[String]]): DeIdentificationParams.this.type
- Definition Classes
- BaseDeidParams
-
def
setStaticObfuscationPairs(pairs: Array[StaticObfuscationEntity]): DeIdentificationParams.this.type
- Definition Classes
- BaseDeidParams
-
def
setStaticObfuscationPairs(pairs: Array[Array[String]]): DeIdentificationParams.this.type
Sets the static obfuscation pairs.
Sets the static obfuscation pairs. Each pair should contain three elements: original, entity type, and fake. The pairs must have exactly 3 elements: [original, entityType, fake].
- pairs
An array of arrays containing the static obfuscation pairs.
- Definition Classes
- BaseDeidParams
-
def
setUnnormalizedDateMode(mode: String): DeIdentificationParams.this.type
The mode to use if the date is not formatted.
The mode to use if the date is not formatted. Options: [mask, obfuscate, skip] Default: obfuscate
- Definition Classes
- BaseDeidParams
-
def
setUseShiftDays(s: Boolean): DeIdentificationParams.this.type
Sets the value of useShiftDays.
Sets the value of useShiftDays. Whether to use the random shift day when the document has this in its metadata. DocumentHashCoder can create 'dateshift' based on the document. Default: false
- Definition Classes
- DeIdentificationParams → BaseDeidParams
- def setZipCodeTag(s: String): DeIdentificationParams.this.type
-
val
staticObfuscationPairs: StructFeature[Array[StaticObfuscationEntity]]
A resource containing static obfuscation pairs.
A resource containing static obfuscation pairs. Each pair should contain three elements: original, entity type, and fake.
- Definition Classes
- BaseDeidParams
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
val
unnormalizedDateMode: Param[String]
The mode to use if the date is not formatted.
The mode to use if the date is not formatted. Options: [mask, obfuscate, skip] Default: obfuscate
- Definition Classes
- BaseDeidParams
-
val
useShifDays: BooleanParam
Use shift days : Whether to use the random shift day when the document has this in its metadata.
Use shift days : Whether to use the random shift day when the document has this in its metadata. Default: False
-
val
useShiftDays: BooleanParam
Whether to use the random shift day when the document has this in its metadata.
Whether to use the random shift day when the document has this in its metadata. DocumentHashCoder can create 'dateshift' based on the document. Default: false
- Definition Classes
- BaseDeidParams
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
- val zipCodeTag: Param[String]
Deprecated Value Members
-
def
setUseShiftDayse(s: Boolean): DeIdentificationParams.this.type
- Annotations
- @deprecated
- Deprecated
deprecated because of typo