Extract Demographic Entities from Oncology Texts

Description

This model extracts demographic information from oncology texts, including age, gender, and smoking status.

Definitions of Predicted Entities:

  • Age: All mention of ages, past or present, related to the patient or with anybody else.
  • Gender: Gender-specific nouns and pronouns (including words such as “him” or “she”, and family members such as “father”).
  • Race_Ethnicity: The race and ethnicity categories include racial and national origin or sociocultural groups.
  • Smoking_Status: All mentions of smoking related to the patient or to someone else.

Predicted Entities

Age, Gender, Race_Ethnicity, Smoking_Status

Live Demo Open in Colab Copy S3 URI

How to use

document_assembler = DocumentAssembler()\
    .setInputCol("text")\
    .setOutputCol("document")

sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
    .setInputCols(["document"])\
    .setOutputCol("sentence")

tokenizer = Tokenizer() \
    .setInputCols(["sentence"]) \
    .setOutputCol("token")

word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical", "en", "clinical/models")\
    .setInputCols(["sentence", "token"]) \
    .setOutputCol("embeddings")                

ner = MedicalNerModel.pretrained("ner_oncology_demographics", "en", "clinical/models") \
    .setInputCols(["sentence", "token", "embeddings"]) \
    .setOutputCol("ner")

ner_converter = NerConverter() \
    .setInputCols(["sentence", "token", "ner"]) \
    .setOutputCol("ner_chunk")

pipeline = Pipeline(stages=[document_assembler,
                            sentence_detector,
                            tokenizer,
                            word_embeddings,
                            ner,
                            ner_converter])

data = spark.createDataFrame([["The patient is a 40-year-old man with history of heavy smoking."]]).toDF("text")

result = pipeline.fit(data).transform(data)
val document_assembler = new DocumentAssembler()
    .setInputCol("text")
    .setOutputCol("document")
    
val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")
    .setInputCols("document")
    .setOutputCol("sentence")
    
val tokenizer = new Tokenizer()
    .setInputCols("sentence")
    .setOutputCol("token")
    
val word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical", "en", "clinical/models")
    .setInputCols(Array("sentence", "token"))
    .setOutputCol("embeddings")                
    
val ner = MedicalNerModel.pretrained("ner_oncology_demographics", "en", "clinical/models")
    .setInputCols(Array("sentence", "token", "embeddings"))
    .setOutputCol("ner")
    
val ner_converter = new NerConverter()
    .setInputCols(Array("sentence", "token", "ner"))
    .setOutputCol("ner_chunk")

        
val pipeline = new Pipeline().setStages(Array(document_assembler,
                            sentence_detector,
                            tokenizer,
                            word_embeddings,
                            ner,
                            ner_converter))    

val data = Seq("The patient is a 40-year-old man with history of heavy smoking.").toDS.toDF("text")

val result = pipeline.fit(data).transform(data)
import nlu
nlu.load("en.med_ner.oncology_demographics").predict("""The patient is a 40-year-old man with history of heavy smoking.""")

Results

| chunk       | ner_label      |
|:------------|:---------------|
| 40-year-old | Age            |
| man         | Gender         |
| smoking     | Smoking_Status |

Model Information

Model Name: ner_oncology_demographics
Compatibility: Healthcare NLP 4.2.2+
License: Licensed
Edition: Official
Input Labels: [sentence, token, embeddings]
Output Labels: [ner]
Language: en
Size: 34.6 MB

References

In-house annotated oncology case reports.

Benchmarking

         label   tp  fp  fn  total  precision  recall   f1
Smoking_Status   60  19   8     68       0.76    0.88 0.82
           Age  934  33  15    949       0.97    0.98 0.97
Race_Ethnicity   57   5   5     62       0.92    0.92 0.92
        Gender 1248  18   6   1254       0.99    1.00 0.99
     macro_avg 2299  75  34   2333       0.91    0.95 0.93
     micro_avg 2299  75  34   2333       0.97    0.99 0.98