Type Members

1.  abstract class BaseFhirDeIdentification extends Transformer with HasFeatures with LightDeIdentificationParams with DeidModelParams with CheckLicense with HasInputCol with HasOutputAnnotationCol with ParamsAndFeaturesWritable
2.  class CdaDeIdentification extends BaseFhirDeIdentification

   A Spark Transformer for de-identifying HL7 CDA (Clinical Document Architecture) documents using javax.xml DOM parsing.

   A Spark Transformer for de-identifying HL7 CDA (Clinical Document Architecture) documents using javax.xml DOM parsing.

   Overview

   Performs field-level and free-text de-identification on CDA R2 XML documents using XPath expressions. Supports both structured field obfuscation and NLP-based free-text de-identification.

   Key Features

   - **XPath-based de-identification**: Target specific CDA elements using dot notation - **Attribute support**: Access XML attributes using @ notation (e.g., "telecom/@value") - **Namespace-aware**: Handles HL7 v3 namespace automatically - **Structured field obfuscation**: Replace names, dates, addresses, phone numbers, IDs - **Free-text de-identification**: Process narrative sections using NLP pipelines

   Path Notation

   Paths support both dot (.) and slash (/) as separators. Attributes can be specified with or without @ prefix. Free-text container paths (e.g. section.text) apply de-identification recursively to all descendant TEXT_NODEs.

   // Element paths (both notations work)
   "recordTarget.patientRole.patient.name.given"
   "recordTarget/patientRole/patient/name/given"
   
   // Attribute paths (all formats work)
   "recordTarget.patientRole.telecom/@value"     // explicit @
   "recordTarget.patientRole.telecom.value"      // auto-detected
   "recordTarget/patientRole/telecom/@value"     // with slashes
   "recordTarget.patientRole.id.extension"       // known attribute
   
   // Nested structures
   "component.structuredBody.component.section.text"
   "component/structuredBody/component/section/text"

   Usage Example

   val deid = new CdaDeIdentification()
     .setInputCol("cda_xml")
     .setOutputCol("deidentified_cda")
     .setMode("obfuscate")
     .setMappingRules(Map(
       "recordTarget.patientRole.patient.name.given" -> "first_name",
       "recordTarget.patientRole.patient.name.family" -> "last_name",
       "recordTarget.patientRole.addr.streetAddressLine" -> "Address",
       "recordTarget.patientRole.telecom/@value" -> "Phone",
       "author.assignedAuthor.assignedPerson.name.given" -> "first_name",
       "custodian.assignedCustodian.representedCustodianOrganization.name" -> "Organization"
     ))
     .setFreeTextPaths(Array(
       "component.structuredBody.component.section.text"
     ))
     .setPipeline(spark, deidPipeline, "deidentified")
   
   val result = deid.deidentify(cdaXmlString)

   Structured Narrative Handling (Tables, Definition Lists)

   When tableHandling is true (the default), structured elements inside free-text blocks are processed with header-aware context so the NLP pipeline receives a longer, more informative input instead of a single short cell. The following structures are recognized:

   • `: column headers (``) are never obfuscated under this mode, which prevents column/row labels from being mutated and improves accuracy on cell values. Set `tableHandling` to `false` to fall back to the legacy recursive obfuscation that walks every descendant text node uniformly. Use `excludeFreeTextTags` to fully skip any additional element by local name (case insensitive) during free-text recursion, e.g. `Array("sup", "footnote")`.

     See also

     HL7 CDA Standard

     BaseFhirDeIdentification for base de-identification functionality

      class FhirDeIdentification extends BaseFhirDeIdentification

     A Spark Transformer for de-identifying FHIR resources according to configurable privacy rules.

     A Spark Transformer for de-identifying FHIR resources according to configurable privacy rules.

     Overview

     Performs field-level obfuscation on FHIR JSON documents using FHIR Path expressions. Supports R4, R5, and DSTU3 FHIR versions with type-aware de-identification strategies. Additionally, supports different parser types (JSON, XML) for FHIR resources.

     Example:

     1. Basic Pipeline Usage

        val deid = new FhirDeIdentification()
          .setInputCol("raw_fhir")
          .setOutputCol("deidentified")
          .setMode("obfuscate")
          .setMappingRules(Map("Patient.birthDate" -> "Date"))
        
        val pipeline = new Pipeline().setStages(Array(deid))

     See also

     FHIR Specification

      class Hl7v2DeIdentification extends BaseFhirDeIdentification

     A Spark Transformer for de-identifying HL7 v2.x messages (the classic pipe / hat delimited "vertical bar" format, e.g.

     A Spark Transformer for de-identifying HL7 v2.x messages (the classic pipe / hat delimited "vertical bar" format, e.g. ADTA01, ORUR01).

     Overview

     Performs field-level obfuscation on HL7 v2 messages using a lightweight, dependency-free delimiter parser. Because the HL7 v2 wire format (segments, fields, components, repetitions, subcomponents and the escape character declared in MSH-1/MSH-2) is identical across all v2.x versions, a single Terser-style path notation targets PHI uniformly from v2.1 through v2.9 without any version-specific structure libraries.

     Path Notation

     "PID-5"       -> segment PID, field 5 (all components / repetitions)
     "PID-5-1"     -> field 5, component 1 (family name)
     "PID-5-1-2"   -> field 5, component 1, subcomponent 2
     "PID(2)-5-1"  -> the 2nd PID segment occurrence, field 5, component 1

     Field numbers match the HL7 specification exactly, including the MSH off-by-one convention (MSH-1 is the field separator, MSH-2 the encoding characters).

     Structured Field De-identification

     val deid = new Hl7v2DeIdentification()
       .setInputCol("hl7")
       .setOutputCol("deidentified")
       .setMode("obfuscate")
       .setMappingRules(Map(
         "PID-5-1" -> "LAST_NAME",   // family name
         "PID-5-2" -> "FIRST_NAME",   // given name
         "PID-7"   -> "DATE",      // date of birth
         "PID-11-1" -> "STREET",
         "PID-13"  -> "PHONE"
       ))
     
     val result = deid.deidentify(hl7MessageString)

     Free-text De-identification

     Narrative fields such as OBX-5 (observation value) and NTE-3 (comment) can be routed through an NLP de-identification pipeline:

     deid
       .setFreeTextPaths(Array("OBX-5", "NTE-3"))
       .setPipeline(spark, deidPipeline, "obfuscated")

     See also

     HL7 v2 Standard

     BaseFhirDeIdentification for the shared de-identification functionality

      trait PretrainedReadableCdaDeIdentification extends ParamsAndFeaturesReadable[CdaDeIdentification] with HasPretrained[CdaDeIdentification]
      trait PretrainedReadableFhirDeIdentification extends ParamsAndFeaturesReadable[FhirDeIdentification] with HasPretrained[FhirDeIdentification]
      trait PretrainedReadableHl7v2DeIdentification extends ParamsAndFeaturesReadable[Hl7v2DeIdentification] with HasPretrained[Hl7v2DeIdentification]

     Value Members

     1.  object CdaDeIdentification extends PretrainedReadableCdaDeIdentification with Serializable

        This is the companion object of CdaDeIdentification.

        This is the companion object of CdaDeIdentification. Please refer to that class for the documentation.

     2.  object FhirDeIdentification extends PretrainedReadableFhirDeIdentification with Serializable

        This is the companion object of FhirDeIdentification.

        This is the companion object of FhirDeIdentification. Please refer to that class for the documentation.

     3.  object FhirParserTypes

        Attributes

        protected

     4.  object FhirUtil

        Attributes

        protected

     5.  object FhirVersions

        Attributes

        protected

     6.  object Hl7v2DeIdentification extends PretrainedReadableHl7v2DeIdentification with Serializable

        This is the companion object of Hl7v2DeIdentification.

        This is the companion object of Hl7v2DeIdentification. Please refer to that class for the documentation.

     Ungrouped

     ` inside the header row) are preserved verbatim and each data cell (`	`) is sent to the pipeline as `" : "`. The obfuscated cell value is then extracted from the pipeline output and written back into the document. When a data row begins with a row-label `	` (e.g. "Blood Pressure"), that label is used as the per-row context instead of the column header. For tables without a strict `	`-only header row, the first row of a multi-row, multi-column table is automatically promoted to column headers. - Inline cell labels: when a `	` (or ` `) starts with a styling element such as `Title:`, that element is treated as the cell's own label (overrides any external header) and is preserved verbatim - only the text after it is obfuscated. - ` ` line splitting: a cell's value text is split on top-level ` ` elements and each line is processed independently, so multi-paragraph narrative cells keep their line layout and never produce concatenated tokens like `Hgb12.5`. - ` ` / ` ` / ` `: each ` ` is processed with its preceding ` ` as context. `	` and ` ` (and the rest of `