Clinical Deidentification (English, Glove, Augmented)

Description

This pipeline is trained with lightweight glove_100d embeddings and can be used to deidentify PHI information from medical texts. The PHI information will be masked and obfuscated in the resulting text. The pipeline can mask and obfuscate AGE, CONTACT, DATE, ID, LOCATION, NAME, PROFESSION, CITY, COUNTRY, DOCTOR, HOSPITAL, IDNUM, MEDICALRECORD, ORGANIZATION, PATIENT, PHONE, PROFESSION, STREET, USERNAME, ZIP, ACCOUNT, LICENSE, VIN, SSN, DLN, PLATE, IPADDR entities.

It’s different to clinical_deidentification_glove in the way it manages PHONE and PATIENT, having apart from the NER, some rules in Contextual Parser components.

Live Demo Open in Colab Copy S3 URI

How to use

from sparknlp.pretrained import PretrainedPipeline

deid_pipeline = PretrainedPipeline("clinical_deidentification_glove_augmented", "en", "clinical/models")

deid_pipeline.annotate("""Record date : 2093-01-13, David Hale, M.D. IP: 203.120.223.13. The driver's license no:A334455B. the SSN: 324598674 and e-mail: hale@gmail.com. Name : Hendrickson, Ora MR. # 719435 Date : 01/13/93. PCP : Oliveira, 25 years old. Record date : 2079-11-09, Patient's VIN : 1HGBH41JXMN109286.""")
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline

val deid_pipeline = PretrainedPipeline("clinical_deidentification_glove_augmented", "en", "clinical/models")

val result = pipeline.annotate("""Record date : 2093-01-13, David Hale, M.D. IP: 203.120.223.13. The driver's license no:A334455B. the SSN: 324598674 and e-mail: hale@gmail.com. Name : Hendrickson, Ora MR. # 719435 Date : 01/13/93. PCP : Oliveira, 25 years old. Record date : 2079-11-09, Patient's VIN : 1HGBH41JXMN109286.""")
import nlu
nlu.load("en.deid.glove_augmented.pipeline").predict("""Record date : 2093-01-13, David Hale, M.D. IP: 203.120.223.13. The driver's license no:A334455B. the SSN: 324598674 and e-mail: hale@gmail.com. Name : Hendrickson, Ora MR. # 719435 Date : 01/13/93. PCP : Oliveira, 25 years old. Record date : 2079-11-09, Patient's VIN : 1HGBH41JXMN109286.""")

Results

{'masked': ['Record date : <DATE>, <DOCTOR>, M.D.',
    'IP: <IPADDR>.',
    "The driver's license no: <LICENSE>.",
    'The SSN: <SSN> and e-mail: <EMAIL>.',
    'Name : <PATIENT> MR. # <MEDICALRECORD> Date : <DATE>.',
    'PCP : <DOCTOR>, <AGE> years old.',
    'Record date : <DATE>, <DOCTOR> : <VIN>.'],
 'masked_fixed_length_chars': ['Record date : ****, ****, M.D.',
    'IP: ****.',
    "The driver's license no: ****.",
    'The SSN: **** and e-mail: ****.',
    'Name : **** MR. # **** Date : ****.',
    'PCP : ****, **** years old.',
    'Record date : ****, **** : ****.'],
 'masked_with_chars': ['Record date : [********], [********], M.D.',
    'IP: [************].',
    "The driver's license no: [******].",
    'The SSN: [*******] and e-mail: [************].',
    'Name : [**************] MR. # [****] Date : [******].',
    'PCP : [******], ** years old.',
    'Record date : [********], [***********] : [***************].'],
 'ner_chunk': ['2093-01-13',
    'David Hale',
    'A334455B',
    '324598674',
    'hale@gmail.com',
    'Hendrickson, Ora',
    '719435',
    '01/13/93',
    'Oliveira',
    '25',
    '2079-11-09',
    "Patient's VIN",
    '1HGBH41JXMN109286'],
 'obfuscated': ['Record date : 2093-01-23, Dr Marshia Curling, M.D.',
    'IP: 004.004.004.004.',
    "The driver's license no: 123XX123.",
    'The SSN: SSN-089-89-9294 and e-mail: Mikey@hotmail.com.',
    'Name : Stephania Chang MR. # E5881795 Date : 02-14-1983.',
    'PCP : Dr Lovella Israel, 52 years old.',
    'Record date : 2079-11-14, Dr Colie Carne : 3CCCC22DDDD333888.'],
 'sentence': ['Record date : 2093-01-13, David Hale, M.D.',
    'IP: 203.120.223.13.',
    "The driver's license no: A334455B.",
    'The SSN: 324598674 and e-mail: hale@gmail.com.',
    'Name : Hendrickson, Ora MR. # 719435 Date : 01/13/93.',
    'PCP : Oliveira, 25 years old.',
    "Record date : 2079-11-09, Patient's VIN : 1HGBH41JXMN109286."]}

Model Information

Model Name: clinical_deidentification_glove_augmented
Type: pipeline
Compatibility: Healthcare NLP 4.1.0+
License: Licensed
Edition: Official
Language: en
Size: 181.3 MB

Included Models

  • DocumentAssembler
  • SentenceDetector
  • TokenizerModel
  • WordEmbeddingsModel
  • MedicalNerModel
  • NerConverter
  • MedicalNerModel
  • NerConverter
  • ChunkMergeModel
  • ContextualParserModel
  • ContextualParserModel
  • ContextualParserModel
  • ContextualParserModel
  • ContextualParserModel
  • ContextualParserModel
  • ContextualParserModel
  • ContextualParserModel
  • ChunkMergeModel
  • ChunkMergeModel
  • DeIdentificationModel
  • DeIdentificationModel
  • DeIdentificationModel
  • DeIdentificationModel
  • Finisher