Clinical Deidentification Pipeline (English)

Description

This pipeline can be used to deidentify PHI information from medical texts. The PHI information will be masked and obfuscated in the resulting text. The pipeline can mask and obfuscate LOCATION, CONTACT, PROFESSION, NAME, DATE, ID, AGE, MEDICALRECORD, ORGANIZATION, HEALTHPLAN, DOCTOR, USERNAME, URL, DEVICE, CITY, ZIP, STATE, PATIENT, COUNTRY, STREET, PHONE, HOSPITAL, EMAIL, IDNUM, BIOID, FAX, SSN, ACCOUNT, DLN, PLATE, VIN, LICENSE entities.

This pipeline is the optimized version of the previous clinical_deidentification pipelines, resulting in significantly improved speed. It returns obfuscated version of the texts as the result and its masked with entity labels version in the metadata.

Predicted Entities

LOCATION, CONTACT, PROFESSION, NAME, DATE, ID, AGE, MEDICALRECORD, ORGANIZATION, HEALTHPLAN, DOCTOR, USERNAME, URL, DEVICE, CITY, ZIP, STATE, PATIENT, COUNTRY, STREET, PHONE, HOSPITAL, EMAIL, IDNUM, BIOID, FAX, SSN, ACCOUNT, DLN, PLATE, VIN, LICENSE

Live Demo Open in Colab Download Copy S3 URI

How to use

from sparknlp.pretrained import PretrainedPipeline

deid_pipeline = PretrainedPipeline("clinical_deidentification", "en", "clinical/models")

text = """Name : Hendrickson, Ora, Record date: 2093-01-13, MR #719435.
Dr. John Green, ID: 1231511863, IP 203.120.223.13.
He is a 60-year-old male was admitted to the Day Hospital for cystectomy on 01/13/93.
Patient's VIN : 1HGBH41JXMN109286, SSN #333-44-6666, Driver's license no: A334455B.
Phone (302) 786-5227, 0295 Keats Street, San Francisco, E-MAIL: smith@gmail.com."""

result = deid_pipeline.fullAnnotate(text)

print('\n'.join([i.metadata['masked'] for i in result[0]['obfuscated']]))
print('\n'.join([i.result for i in result[0]['obfuscated']]))

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline

val deid_pipeline = PretrainedPipeline("clinical_deidentification", "en", "clinical/models")

val text = """Name : Hendrickson, Ora, Record date: 2093-01-13, MR #719435.
Dr. John Green, ID: 1231511863, IP 203.120.223.13.
He is a 60-year-old male was admitted to the Day Hospital for cystectomy on 01/13/93.
Patient's VIN : 1HGBH41JXMN109286, SSN #333-44-6666, Driver's license no: A334455B.
Phone (302) 786-5227, 0295 Keats Street, San Francisco, E-MAIL: smith@gmail.com."""

val result = deid_pipeline.fullAnnotate(text)

println(deid_result(0)("obfuscated").map(_("metadata")("masked").toString).mkString("\n"))
println(deid_result(0)("obfuscated").map(_("result").toString).mkString("\n"))

Results

Masked with entity labels
------------------------------
Name : <PATIENT>, Record date: <DATE>, MR <ID>.
Dr. <DOCTOR>, ID: <DEVICE>, IP <IPADDR>.
He is a <AGE>-year-old male was admitted to the <HOSPITAL> for cystectomy on <DATE>.
Patient's VIN : <VIN>, SSN <SSN>, Driver's license no: <DLN>.
Phone <PHONE>, <STREET>, <CITY>, E-MAIL: <EMAIL>.

Obfuscated
------------------------------
Name : Lucretia Kern, Record date: 2093-02-16, MR #169678.
Dr. Ramon Dredge, ID: U4680041, IP 005.005.005.005.
He is a 79-year-old male was admitted to the ST. JOSEPH REGIONAL MEDICAL CENTER for cystectomy on 02/16/93.
Patient's VIN : 9FYBO17PZWC585277, SSN #824-23-5361, Driver's license no: W431540G.
Phone (867) 619-5093, 800 Share Drive, Wilburton, E-MAIL: Rufus@google.com.

Model Information

Included Models

• DocumentAssembler
• SentenceDetectorDLModel
• TokenizerModel
• WordEmbeddingsModel
• MedicalNerModel
• NerConverter
• MedicalNerModel
• NerConverter
• ChunkMergeModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• TextMatcherInternalModel
• ContextualParserModel
• RegexMatcherModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ChunkMergeModel
• ChunkMergeModel
• DeIdentificationModel
• Finisher