Clinical Deidentification Pipeline (English)

Description

This pipeline can be used to deidentify PHI information from medical texts. The PHI information will be masked and obfuscated in the resulting text. The pipeline can mask and obfuscate ACCOUNT, AGE, BIOID, CITY, CONTACT, COUNTRY, DATE, DEVICE, DLN, DOCTOR, EMAIL, FAX, HEALTHPLAN, HOSPITAL, ID, IPADDR, LICENSE, LOCATION, MEDICALRECORD, NAME, ORGANIZATION, PATIENT, PHONE, PLATE, PROFESSION, SREET, SSN, STATE, STREET, URL, USERNAME, VIN, ZIP entities.

Predicted Entities

ACCOUNT, AGE, BIOID, CITY, CONTACT, COUNTRY, City, DATE, DEVICE, DLN, DOCTOR, EMAIL, FAX, HEALTHPLAN, HOSPITAL, ID, IDNUM, IP, LICENSE, LOCATION, LOCATION-OTHER, LOCATION_OTHER, MEDICALRECORD, NAME, ORGANIZATION, PATIENT, PHONE, PLATE, PROFESSION, SSN, STATE, STREET, URL, USERNAME, VIN, ZIP

Live Demo Open in Colab Download Copy S3 URI

How to use

from sparknlp.pretrained import PretrainedPipeline

deid_pipeline = PretrainedPipeline("clinical_deidentification_docwise_wip", "en", "clinical/models")

text = """Name : Hendrickson, Ora, Record date: 2093-01-13, MR: 87719435.
ID: #12315112, Dr. John Green, IP 203.120.223.13.
He is a 60-year-old male was admitted to the Day Hospital for cystectomy on 01/13/93.
Patient's VIN : 1HGBH41JXMN109286, SSN #333-44-6666, Driver's license no: A334455B.
Phone (302) 786-5227, 0295 Keats Street, San Francisco,  CA 94108. E-MAIL: smith@gmail.com."""

deid_result = deid_pipeline.fullAnnotate(text)

print(''.join([i.metadata['masked'] for i in deid_result[0]['obfuscated']]))
print(''.join([i.result for i in deid_result[0]['obfuscated']]))

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline

val deid_pipeline = PretrainedPipeline("clinical_deidentification_docwise_wip", "en", "clinical/models")

val text = """Name : Hendrickson, Ora, Record date: 2093-01-13, MR: 87719435.
ID: #12315112, Dr. John Green, IP 203.120.223.13.
He is a 60-year-old male was admitted to the Day Hospital for cystectomy on 01/13/93.
Patient's VIN : 1HGBH41JXMN109286, SSN #333-44-6666, Driver's license no: A334455B.
Phone (302) 786-5227, 0295 Keats Street, San Francisco,  CA 94108. E-MAIL: smith@gmail.com."""

val deid_result = deid_pipeline.fullAnnotate(text)

println(deid_result(0)("obfuscated").map(_("metadata")("masked").toString).mkString(""))
println(deid_result(0)("obfuscated").map(_("result").toString).mkString(""))

Results

Masked with entity labels
------------------------------
Name : <PATIENT>, Record date: <DATE>, MR: <MEDICALRECORD>.
ID: <IDNUM>, Dr. <DOCTOR>, IP <IPADDR>.
He is a <AGE>-year-old male was admitted to the <HOSPITAL> for cystectomy on <DATE>.
Patient's VIN : <VIN>, SSN <SSN>, Driver's license no: <DLN>.
Phone <PHONE>, <STREET>, <CITY>,  <STATE> <ZIP>.
E-MAIL: <EMAIL>.

Obfuscated
------------------------------
Name : Axel Bohr, Record date: 2093-02-01, MR: 61443154.
ID: #00867619, Dr. Rickard Charles, IP 002.002.002.002.
He is a 73-year-old male was admitted to the LOMA LINDA UNIVERSITY MEDICAL CENTER-MURRIETA for cystectomy on 02/01/93.
Patient's VIN : 5KDTO67TIWP809983, SSN #382-50-5397, Driver's license no: Q734193X.
Phone (902) 409-7353, 1555 Long Pond Road, Pomeroy,  Maryland 29924.
 E-MAIL: Halit@google.com.

Model Information

Included Models

• DocumentAssembler
• InternalDocumentSplitter
• TokenizerModel
• WordEmbeddingsModel
• MedicalNerModel
• MedicalNerModel
• MedicalNerModel
• NerConverterInternalModel
• NerConverterInternalModel
• NerConverterInternalModel
• ChunkMergeModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• TextMatcherInternalModel
• TextMatcherInternalModel
• TextMatcherInternalModel
• ContextualParserModel
• RegexMatcherInternalModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• RegexMatcherInternalModel
• RegexMatcherInternalModel
• RegexMatcherInternalModel
• TextMatcherInternalModel
• ChunkMergeModel
• ChunkMergeModel
• LightDeIdentification
• LightDeIdentification