Clinical Deidentification Pipeline (Document Wise - Benchmark)

Description

This pipeline can be used to deidentify PHI information from medical texts. The PHI information will be masked and obfuscated in the resulting text. The pipeline can mask and obfuscate NAME, IDNUM, CONTACT, LOCATION, AGE, DATE entities. This pipeline is prepared for benchmarking with cloud providers.

Predicted Entities

NAME, IDNUM, CONTACT, LOCATION, AGE, DATE

Live Demo Open in Colab Download Copy S3 URI

How to use

from sparknlp.pretrained import PretrainedPipeline

deid_pipeline = PretrainedPipeline("clinical_deidentification_docwise_benchmark", "en", "clinical/models")               

deid_result = deid_pipeline.fullAnnotate("""Name : Hendrickson, Ora, Record date: 2093-01-13, # 719435.
Dr. John Green, ID: 1231511863, IP 203.120.223.13.
He is a 60-year-old male was admitted to the Day Hospital for cystectomy on 01/13/93.
Patient's VIN : 1HGBH41JXMN109286, SSN #333-44-6666, Driver's license no:A334455B.
Phone (302) 786-5227, Keats Street, San Francisco, E-MAIL: smith@gmail.com.""")

print(''.join([i.result for i in deid_result['mask_entity']]))
print(''.join([i.result for i in deid_result['obfuscated']]))

deid_pipeline = nlp.PretrainedPipeline("clinical_deidentification_docwise_benchmark", "en", "clinical/models")               

deid_result = deid_pipeline.fullAnnotate("""Name : Hendrickson, Ora, Record date: 2093-01-13, # 719435.
Dr. John Green, ID: 1231511863, IP 203.120.223.13.
He is a 60-year-old male was admitted to the Day Hospital for cystectomy on 01/13/93.
Patient's VIN : 1HGBH41JXMN109286, SSN #333-44-6666, Driver's license no:A334455B.
Phone (302) 786-5227, Keats Street, San Francisco, E-MAIL: smith@gmail.com.""")

print(''.join([i.result for i in deid_result['mask_entity']]))
print(''.join([i.result for i in deid_result['obfuscated']]))

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline

val deid_pipeline = PretrainedPipeline("clinical_deidentification_docwise_benchmark", "en", "clinical/models")               

val deid_result = deid_pipeline.fullAnnotate("""Name : Hendrickson, Ora, Record date: 2093-01-13, # 719435.
Dr. John Green, ID: 1231511863, IP 203.120.223.13.
He is a 60-year-old male was admitted to the Day Hospital for cystectomy on 01/13/93.
Patient's VIN : 1HGBH41JXMN109286, SSN #333-44-6666, Driver's license no:A334455B.
Phone (302) 786-5227, Keats Street, San Francisco, E-MAIL: smith@gmail.com.""")

println(deid_result("mask_entity").map(_("result").toString).mkString(""))
println(deid_result("obfuscated").map(_("result").toString).mkString(""))

Results

Masked with entity labels
------------------------------
Name : <NAME>, Record date: <DATE>, # <IDNUM>.
Dr. <NAME>, ID: <IDNUM>, IP <IDNUM>.
He is a <AGE> male was admitted to the <LOCATION> for cystectomy on <DATE>.
Patient's VIN : <IDNUM>, SSN <IDNUM>, Driver's license <IDNUM>.
Phone <CONTACT>, <LOCATION>, <LOCATION>, E-MAIL: <CONTACT>.


Obfuscated
------------------------------
Name : Driscilla George, Record date: 2093-01-19, # 042968.
Dr. Donneta Gaines, ID: 6986066538, IP 978.697.998.68.
He is a 68-year-old male was admitted to the 401 Kendall Drive for cystectomy on 01/19/93.
Patient's VIN : 1NYTN61LXSP129408, SSN #999-22-4444, Driver's license EF:F669988C.
Phone (768) 142-9881, 1141 North Monroe Drive, 1165 Montgomery Drive, E-MAIL: THDYC@NHLDG.JFH.

Model Information

Included Models

• DocumentAssembler
• InternalDocumentSplitter
• TokenizerModel
• WordEmbeddingsModel
• MedicalNerModel
• NerConverterInternalModel
• MedicalNerModel
• MedicalNerModel
• MedicalNerModel
• NerConverterInternalModel
• NerConverterInternalModel
• NerConverterInternalModel
• PretrainedZeroShotNER
• NerConverterInternalModel
• MedicalNerModel
• NerConverterInternalModel
• ContextualEntityRuler
• ChunkMergeModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• TextMatcherInternalModel
• TextMatcherInternalModel
• ContextualParserModel
• RegexMatcherInternalModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• ContextualParserModel
• RegexMatcherInternalModel
• RegexMatcherInternalModel
• ChunkMergeModel
• ChunkMergeModel
• ContextualEntityRuler
• ChunkMergeModel
• LightDeIdentification
• LightDeIdentification