DeIdentificationModel

Companion object DeIdentificationModel

class DeIdentificationModel extends AnnotatorModel[DeIdentificationModel] with DeIdentificationParams with HasSimpleAnnotate[DeIdentificationModel] with Licensed

Contains all the parameters to transform a dataset with three Input Annotations of types DOCUMENT, TOKEN and CHUNK, into its DeIdentified version of by either masking or obfuscating the given CHUNKS.

To create an configured DeIdentificationModel, please see the example of DeIdentification.

See also: DeIdentification to train your own model

Linear Supertypes

Licensed, HasSimpleAnnotate[DeIdentificationModel], DeIdentificationParams, AnnotatorModel[DeIdentificationModel], CanBeLazy, RawAnnotator[DeIdentificationModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[DeIdentificationModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any

Ordering

Grouped
Alphabetic
By Inheritance

Inherited

DeIdentificationModel
Licensed
HasSimpleAnnotate
DeIdentificationParams
AnnotatorModel
CanBeLazy
RawAnnotator
HasOutputAnnotationCol
HasInputAnnotationCols
HasOutputAnnotatorType
ParamsAndFeaturesWritable
HasFeatures
DefaultParamsWritable
MLWritable
Model
Transformer
PipelineStage
Logging
Params
Serializable
Serializable
Identifiable
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new DeIdentificationModel()
new DeIdentificationModel(uid: String)
uid
a unique identifier for the instanced AnnotatorModel

Type Members

type AnnotationContent = Seq[Row]

Attributes
protected
Definition Classes
AnnotatorModel
type AnnotatorType = String

Definition Classes
HasOutputAnnotatorType
implicit class StringReplacement extends AnyRef

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def $[T](param: Param[T]): T

Attributes
protected
Definition Classes
Params
def $$[T](feature: StructFeature[T]): T

Attributes
protected
Definition Classes
HasFeatures
def $$[K, V](feature: MapFeature[K, V]): Map[K, V]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: SetFeature[T]): Set[T]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: ArrayFeature[T]): Array[T]

Attributes
protected
Definition Classes
HasFeatures
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def afterAnnotate(dataset: DataFrame): DataFrame

Definition Classes
DeIdentificationModel → AnnotatorModel
val allTerms: MapFeature[String, List[String]]
dictionary, which contains all terms for using later in anonimization function
def annotate(annotations: Seq[Annotation]): Seq[Annotation]
annotations
The annotations per row that we need to obfuscate the document. Annotations should be DOCUMENT, TOKEN, CHUNK. The annotations of kind TOKEN or CHUNK will be have sentence number in the metadata in any of the annotations of kind Document. If the TOKEN or CHUNK have a sentence number in metadata longer that the sentence number on the document annotations the annotator should throw and exception
returns
The annotations of kind Document masked or obfuscated.

Definition Classes
DeIdentificationModel → HasSimpleAnnotate
final def asInstanceOf[T0]: T0

Definition Classes
Any
def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
This method represents the pipeline method which calls each method one by one It utilizes the main point of interest which is getAnonymizeSentence() and calls it for each sentence
This method represents the pipeline method which calls each method one by one It utilizes the main point of interest which is getAnonymizeSentence() and calls it for each sentence
returns
a Sequence of Anonimized Annotations

Definition Classes
DeIdentificationModel → AnnotatorModel
final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

Attributes
protected
Definition Classes
HasInputAnnotationCols
def chunkFlexibleEquals(chunkA: Annotation, chunkB: Annotation): Boolean
def chunkSameEntity(chunkA: Annotation, chunkB: Annotation): Boolean
final def clear(param: Param[_]): DeIdentificationModel.this.type

Definition Classes
Params
def clone(): AnyRef

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
val consistentObfuscation: BooleanParam
Whether to replace very similar entities in a document with the same randomized term (default: true) The similarity is based on the Levenshtein Distance between the words.
Whether to replace very similar entities in a document with the same randomized term (default: true) The similarity is based on the Levenshtein Distance between the words.

Definition Classes
DeIdentificationParams
def copy(extra: ParamMap): DeIdentificationModel

Definition Classes
RawAnnotator → Model → Transformer → PipelineStage → Params
def copyValues[T <: Params](to: T, extra: ParamMap): T

Attributes
protected
Definition Classes
Params
def createAnonymizeAnnotation(anonymizeSentence: Sentence, offset: Int, idx: Int): Annotation
The method that takes anonymized sentence to create proper Annotation
The method that takes anonymized sentence to create proper Annotation
anonymizeSentence
a sentence, which is anonymized
idx
a index of the sentence
returns
a proper Annotation instance
val dateFormats: StringArrayParam
Format of dates to displace
Format of dates to displace

Definition Classes
DeIdentificationParams
val dateTag: Param[String]
Tag representing what are the NER entity (default: DATE)
Tag representing what are the NER entity (default: DATE)

Definition Classes
DeIdentificationParams
val dateToYear: BooleanParam
true if dates must be converted to years, false otherwise
true if dates must be converted to years, false otherwise

Definition Classes
DeIdentificationParams
val days: IntParam
Number of days to obfuscate the dates by displacement.
Number of days to obfuscate the dates by displacement. If not provided a random integer between 1 and 60 will be used

Definition Classes
DeIdentificationParams
final def defaultCopy[T <: Params](extra: ParamMap): T

Attributes
protected
Definition Classes
Params
def dfAnnotate: UserDefinedFunction

Definition Classes
HasSimpleAnnotate
def displaceMappings(annotation: Annotation, offset: Int): Annotation
The method that takes Mapping and displace de begin and end based in the of set
The method that takes Mapping and displace de begin and end based in the of set
annotation
a sentence, which is anonymized
offset
a index of the sentence
returns
a proper Annotation instance
def duplicateClean(entities: Seq[Annotation], entitiesToCompare: Seq[Annotation]): Seq[Annotation]
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def explainParam(param: Param[_]): String

Definition Classes
Params
def explainParams(): String

Definition Classes
Params
def extraValidate(structType: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
def extraValidateMsg: String

Attributes
protected
Definition Classes
RawAnnotator
final def extractParamMap(): ParamMap

Definition Classes
Params
final def extractParamMap(extra: ParamMap): ParamMap

Definition Classes
Params
val features: ArrayBuffer[Feature[_, _, _]]

Definition Classes
HasFeatures
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]

Definition Classes
DeIdentificationModel → HasFeatures
def get[T](feature: StructFeature[T]): Option[T]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: SetFeature[T]): Option[Set[T]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: ArrayFeature[T]): Option[Array[T]]

Attributes
protected
Definition Classes
HasFeatures
final def get[T](param: Param[T]): Option[T]

Definition Classes
Params
def getAllTerms: Map[String, List[String]]
dictionary, which contains all terms for using later in anonimization function
def getAnonymizeSentence(sentence: String, protectedEntities: Seq[Annotation], dateTag: String = "DATE", days: Int = 0, dateToYear: Boolean = false, minYear: Int = 1929): (String, Seq[Annotation])
Main point of interest.
Main point of interest. This method projects the sentence into the anonymized form This method is called for each sentence in the input collection of Annotations
sentence
a sentence, which we want to anonymize
protectedEntities
a sequence of Entities which we want to anonymize
dateTag
a String which represents the value with which we replace dates
days
a Int which represents how many days back we look at
dateToYear
a flag whether we use displaceDate() or apply dateToYear() method
minYear
a minimum date from which all obfuscated date start, default is 1929
returns
a String, which represents an anonymized sentence
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@native()
final def getDefault[T](param: Param[T]): Option[T]

Definition Classes
Params
def getFakersEntity(entity: String): Seq[String]
def getInputCols: Array[String]

Definition Classes
HasInputAnnotationCols
def getLazyAnnotator: Boolean

Definition Classes
CanBeLazy
def getNerEntitiesBySentence(annotations: Seq[Annotation], sentenceCount: Int): Seq[Seq[Annotation]]
Returns the NER Annotations for each Annotation instance in the input Sequence
Returns the NER Annotations for each Annotation instance in the input Sequence
annotations
a Sequence of Annotation instances
returns
a Sequence of Sequence[IndexedToken], each Sequence represents tokens from each input Annotation
final def getOrDefault[T](param: Param[T]): T

Definition Classes
Params
final def getOutputCol: String

Definition Classes
HasOutputAnnotationCol
def getParam(paramName: String): Param[Any]

Definition Classes
Params
def getRegexEntities(tokensSentence: Seq[IndexedToken], idx: Int): Seq[Annotation]
Returns the Regex Annotations for each IndexedToken in the input Sequence
Returns the Regex Annotations for each IndexedToken in the input Sequence
tokensSentence
a Sequence of IndexedToken instances
returns
a Sequence of Annotation, each Annotation represents Regex Entity
def getRegexPatternsDictionary: Map[String, Array[String]]
dictionary with regular expression patterns that match some protected entity
def getSeed(): Int
def getSentences(annotations: Seq[Annotation]): Seq[Sentence]
Returns the content of each sentence inside the input sequence
Returns the content of each sentence inside the input sequence
annotations
a Sequence of Annotation instances, to return content from
returns
a Sequence of Sentence
def getTokensBySentence(annotations: Seq[Annotation]): Seq[Seq[IndexedToken]]
Returns the tokens for each Annotation instance in the input Sequence
Returns the tokens for each Annotation instance in the input Sequence
annotations
a Sequence of Annotation instances
returns
a Sequence of Sequence[IndexedToken], each Sequence represents tokens from each input Annotation
def handleEntitiesDifferences(AEntities: Seq[Annotation], BEntities: Seq[Annotation]): Seq[Annotation]
Returns a complement of A entities against B entities
Returns a complement of A entities against B entities
AEntities
a sequence of Entities to combine
BEntities
an sequence of Entities to combine
returns
a Sequence of Annotation, which is difference between NER and RegEx
final def hasDefault[T](param: Param[T]): Boolean

Definition Classes
Params
def hasParam(paramName: String): Boolean

Definition Classes
Params
def hasParent: Boolean

Definition Classes
Model
def hashCode(): Int

Definition Classes
AnyRef → Any
Annotations
@native()
val ignoreRegex: BooleanParam
Select if you want to use regex file loaded in the model.
Select if you want to use regex file loaded in the model. If true the default regex file will be not used The default value is false.

Definition Classes
DeIdentificationParams
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
val inputAnnotatorTypes: Array[AnnotatorType]
Input annotator type: DOCUMENT, TOKEN, CHUNK
Input annotator type: DOCUMENT, TOKEN, CHUNK

Definition Classes
DeIdentificationModel → HasInputAnnotationCols
final val inputCols: StringArrayParam

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def isDefined(param: Param[_]): Boolean

Definition Classes
Params
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val isRandomDateDisplacement: BooleanParam
Use a random displacement days in dates entities,that random number is based on the DeIdentificationParams.seed If true use random displacement days in dates entities,if false use the DeIdentificationParams.days The default value is false.
Use a random displacement days in dates entities,that random number is based on the DeIdentificationParams.seed If true use random displacement days in dates entities,if false use the DeIdentificationParams.days The default value is false.

Definition Classes
DeIdentificationParams
def isRegexMatch(token: String, regexPatterns: Array[String]): Boolean
Returns Boolean flag, which says if the token matches at least one pattern from array
Returns Boolean flag, which says if the token matches at least one pattern from array
token
a token of interest to check for the match
regexPatterns
an Array of String to check against the token
returns
a Boolean flag, representing if the token matches at least pattern one of regexPatterns
final def isSet(param: Param[_]): Boolean

Definition Classes
Params
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
val lazyAnnotator: BooleanParam

Definition Classes
CanBeLazy
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
val mappingsColumn: Param[String]
This is the mapping column that will return the Annotations chunks with the fake entities
This is the mapping column that will return the Annotations chunks with the fake entities

Definition Classes
DeIdentificationParams
def mergeEntities(nerEntities: Seq[Annotation], regexEntities: Seq[Annotation], regexOverride: Boolean = false): Seq[Annotation]
Returns a combined Sequence of Annotations, cleaned from duplicates
Returns a combined Sequence of Annotations, cleaned from duplicates
nerEntities
a sequence of NER Entities to combine
regexEntities
an sequence of Regex Entities to combine
returns
a Sequence of Annotation, which is result of a merge without duplicates
val minYear: IntParam
Minimum year to use when converting date to year
Minimum year to use when converting date to year

Definition Classes
DeIdentificationParams
val mode: Param[String]
Mode for Anonymizer ['mask'|'obfuscate'] Given the following text
Mode for Anonymizer ['mask'|'obfuscate'] Given the following text
"David Hale visited EEUU a couple of years ago"
Mask mode: The entities will be replaced by their entity types. Example "<PERSON> visited <COUNTRY> a couple of years ago"
Obfuscate mode:
The entity is replaced by a obfuscator term:
"Bryan Johnson visited Japon a couple of years ago"

Definition Classes
DeIdentificationParams
def msgHelper(schema: StructType): String

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@native()
val obfuscateDate: BooleanParam
When mode=="obfuscate" whether to obfuscate dates or not.
When mode=="obfuscate" whether to obfuscate dates or not. This param helps in consistency to make dateFormats more visible. When setting to true, make sure dateFormats param fits the needs (default: false) WHen setting to 'false' then the date will be mask to <DATE>

Definition Classes
DeIdentificationParams
val obfuscateRefSource: Param[String]
The source of obfuscation of to obfuscate the entities.For dates entities doesnt apply tha method.
The source of obfuscation of to obfuscate the entities.For dates entities doesnt apply tha method. The values ar the following: 'file': Takes the entities from the obfuscatorRefFile 'faker': Takes the entities from the Faker module 'both': Takes the entities from the obfuscatorRefFile and the faker module randomly.

Definition Classes
DeIdentificationParams
def onWrite(path: String, spark: SparkSession): Unit

Attributes
protected
Definition Classes
ParamsAndFeaturesWritable
val outputAnnotatorType: AnnotatorType
Output annotator types: DOCUMENT
Output annotator types: DOCUMENT

Definition Classes
DeIdentificationModel → HasOutputAnnotatorType
final val outputCol: Param[String]

Attributes
protected
Definition Classes
HasOutputAnnotationCol
lazy val params: Array[Param[_]]

Definition Classes
Params
var parent: Estimator[DeIdentificationModel]

Definition Classes
Model
val regexOverride: BooleanParam
If is true prioritize the regex entities, if is false prioritize the ner.
If is true prioritize the regex entities, if is false prioritize the ner. The default value is false.

Definition Classes
DeIdentificationParams
val regexPatternsDictionary: MapFeature[String, Array[String]]
dictionary with regular expression patterns that match some protected entity
def replaceRegExFlavors(word: String): String
This is simple RegEx replace which removes some punctuation tokens from input
This is simple RegEx replace which removes some punctuation tokens from input
word
a String, inside which we want to replace flavors
returns
a String, which represents a cleaned version
val sameEntityThreshold: DoubleParam
Similarity threshold [0.0-1.0] to consider two appearances of an entity as the same (default: 0.9) For date entities this method doesn't apply.
Similarity threshold [0.0-1.0] to consider two appearances of an entity as the same (default: 0.9) For date entities this method doesn't apply.

Definition Classes
DeIdentificationParams
def save(path: String): Unit

Definition Classes
MLWritable
Annotations
@Since( "1.6.0" ) @throws( ... )
val seed: IntParam
It is the seed to select the entities on obfuscate mode.With the seed you can reply a execution several times with the same ouptut.
It is the seed to select the entities on obfuscate mode.With the seed you can reply a execution several times with the same ouptut.

Definition Classes
DeIdentificationParams
def set[T](feature: StructFeature[T], value: T): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: SetFeature[T], value: Set[T]): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: ArrayFeature[T], value: Array[T]): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def set(paramPair: ParamPair[_]): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
Params
final def set(param: String, value: Any): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
Params
final def set[T](param: Param[T], value: T): DeIdentificationModel.this.type

Definition Classes
Params
def setAllTerms(value: Map[String, List[String]]): DeIdentificationModel.this.type
dictionary, which contains all terms for using later in anonimization function
def setConsistentObfuscation(s: Boolean): DeIdentificationModel.this.type
Whether to replace very similar entities in a document with the same randomized term (default: true) The similarity is based on the Levenshtein Distance between the words.
Whether to replace very similar entities in a document with the same randomized term (default: true) The similarity is based on the Levenshtein Distance between the words.

Definition Classes
DeIdentificationParams
def setDateFormats(s: Array[String]): DeIdentificationModel.this.type
Format of dates to displace
Format of dates to displace

Definition Classes
DeIdentificationParams
def setDateTag(s: String): DeIdentificationModel.this.type
Tag representing what are the NER entity (default: DATE)
Tag representing what are the NER entity (default: DATE)

Definition Classes
DeIdentificationParams
def setDateToYear(s: Boolean): DeIdentificationModel.this.type
true if dates must be converted to years, false otherwise
true if dates must be converted to years, false otherwise

Definition Classes
DeIdentificationParams
def setDays(k: Int): DeIdentificationModel.this.type
Number of days to obfuscate the dates by displacement.
Number of days to obfuscate the dates by displacement. If not provided a random integer between 1 and 60 will be used

Definition Classes
DeIdentificationParams
def setDefault[T](feature: StructFeature[T], value: () ⇒ T): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def setDefault(paramPairs: ParamPair[_]*): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
Params
final def setDefault[T](param: Param[T], value: T): DeIdentificationModel.this.type

Attributes
protected
Definition Classes
Params
def setIgnoreRegex(s: Boolean): DeIdentificationModel.this.type
Select if you want to use regex file loaded in the model.
Select if you want to use regex file loaded in the model. If true the default regex file will be not used The default value is false.

Definition Classes
DeIdentificationParams
final def setInputCols(value: String*): DeIdentificationModel.this.type

Definition Classes
HasInputAnnotationCols
final def setInputCols(value: Array[String]): DeIdentificationModel.this.type

Definition Classes
HasInputAnnotationCols
def setIsRandomDateDisplacement(s: Boolean): DeIdentificationModel.this.type
Use a random displacement days in dates entities,that random number is based on the DeIdentificationParams.seed If true use random displacement days in dates entities, if false use the DeIdentificationParams.days The default value is false.
Use a random displacement days in dates entities,that random number is based on the DeIdentificationParams.seed If true use random displacement days in dates entities, if false use the DeIdentificationParams.days The default value is false.

Definition Classes
DeIdentificationParams
def setLazyAnnotator(value: Boolean): DeIdentificationModel.this.type

Definition Classes
CanBeLazy
def setMappingsColumn(s: String): DeIdentificationModel.this.type
This is the mapping column that will return the Annotations chunks with the fake entities
This is the mapping column that will return the Annotations chunks with the fake entities

Definition Classes
DeIdentificationParams
def setMinYear(s: Int): DeIdentificationModel.this.type
Minimum year to use when converting date to year
Minimum year to use when converting date to year

Definition Classes
DeIdentificationParams
def setMode(m: String): DeIdentificationModel.this.type
Mode for Anonymizer ['mask'|'obfuscate'] Given the following text
Mode for Anonymizer ['mask'|'obfuscate'] Given the following text
"David Hale visited EEUU a couple of years ago"
Mask mode: The entities will be replaced by their entity types. Example "<PERSON> visited <COUNTRY> a couple of years ago"
Obfuscate mode:
The entity is replaced by an obfuscator's term:
"Bryan Johnson visited Japon a couple of years ago"

Definition Classes
DeIdentificationParams
def setObfuscateDate(s: Boolean): DeIdentificationModel.this.type
When mode=="obfuscate" whether to obfuscate dates or not.
When mode=="obfuscate" whether to obfuscate dates or not. This param helps in consistency to make dateFormats more visible. When setting to true, make sure dateFormats param fits the needs (default: false) WHen setting to 'false' then the date will be mask to <DATE>

Definition Classes
DeIdentificationParams
def setObfuscateRefSource(s: String): DeIdentificationModel.this.type
The source of obfuscation of to obfuscate the entities.For dates entities doesnt apply tha method.
The source of obfuscation of to obfuscate the entities.For dates entities doesnt apply tha method. The values ar the following: 'file': Takes the entities from the obfuscatorRefFile 'faker': Takes the entities from the Faker module 'both': Takes the entities from the obfuscatorRefFile and the faker module randomly.

Definition Classes
DeIdentificationParams
final def setOutputCol(value: String): DeIdentificationModel.this.type

Definition Classes
HasOutputAnnotationCol
def setParent(parent: Estimator[DeIdentificationModel]): DeIdentificationModel

Definition Classes
Model
def setRegexOverride(s: Boolean): DeIdentificationModel.this.type
If true prioritize the regex if false prioritize the ner.
If true prioritize the regex if false prioritize the ner. The default value is false.

Definition Classes
DeIdentificationParams
def setRegexPatternsDictionary(value: Map[String, Array[String]]): DeIdentificationModel.this.type
dictionary with regular expression patterns that match some protected entity
def setSameEntityThreshold(s: Double): DeIdentificationModel.this.type
Similarity threshold [0.0-1.0] to consider two appearances of an entity as the same (default: 0.9) For date entities this method doesn't apply.
Similarity threshold [0.0-1.0] to consider two appearances of an entity as the same (default: 0.9) For date entities this method doesn't apply.

Definition Classes
DeIdentificationParams
def setSeed(s: Int): DeIdentificationModel.this.type
It is the seed to select the entities on obfuscate mode.With the seed you can reply a execution several times with the same ouptut.
It is the seed to select the entities on obfuscate mode.With the seed you can reply a execution several times with the same ouptut.

Definition Classes
DeIdentificationParams
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Identifiable → AnyRef → Any
final def transform(dataset: Dataset[_]): DataFrame

Definition Classes
AnnotatorModel → Transformer
def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" )
def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" ) @varargs()
final def transformSchema(schema: StructType): StructType

Definition Classes
RawAnnotator → PipelineStage
def transformSchema(schema: StructType, logging: Boolean): StructType

Attributes
protected
Definition Classes
PipelineStage
Annotations
@DeveloperApi()
def udfDocuments: UserDefinedFunction
def udfProtectedEntities: UserDefinedFunction
val uid: String

Definition Classes
DeIdentificationModel → Identifiable
def validate(schema: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
def wrapColumn(col: Column): Column
def wrapColumnMetadata(col: Column): Column

Attributes
protected
Definition Classes
RawAnnotator
def write: MLWriter

Definition Classes
ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable

Packages

DeIdentificationModel

Companion object DeIdentificationModel

class DeIdentificationModel extends AnnotatorModel[DeIdentificationModel] with DeIdentificationParams with HasSimpleAnnotate[DeIdentificationModel] with Licensed

Instance Constructors

Type Members

Value Members

Inherited from Licensed

Inherited from HasSimpleAnnotate[DeIdentificationModel]

Inherited from DeIdentificationParams

Inherited from AnnotatorModel[DeIdentificationModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[DeIdentificationModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[DeIdentificationModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

Parameter getters

Packages

DeIdentificationModel 

Companion object DeIdentificationModel

class DeIdentificationModel extends AnnotatorModel[DeIdentificationModel] with DeIdentificationParams with HasSimpleAnnotate[DeIdentificationModel] with Licensed

Instance Constructors

Type Members

Value Members

Inherited from Licensed

Inherited from HasSimpleAnnotate[DeIdentificationModel]

Inherited from DeIdentificationParams

Inherited from AnnotatorModel[DeIdentificationModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[DeIdentificationModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[DeIdentificationModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

Parameter getters

DeIdentificationModel