Description
This model extracts demographic information from oncology texts, including age, gender and smoking status.
Definitions of Predicted Entities:
Age: All mention of ages, past or present, related to the patient or with anybody else.Gender: Gender-specific nouns and pronouns (including words such as “him” or “she”, and family members such as “father”).Race_Ethnicity: The race and ethnicity categories include racial and national origin or sociocultural groups.Smoking_Status: All mentions of smoking related to the patient or to someone else.
Predicted Entities
Age, Gender, Race_Ethnicity, Smoking_Status
Live Demo Open in Colab Copy S3 URI
How to use
document_assembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer() \
.setInputCols(["sentence"]) \
.setOutputCol("token")
word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical", "en", "clinical/models")\
.setInputCols(["sentence", "token"]) \
.setOutputCol("embeddings")
ner = MedicalNerModel.pretrained("ner_oncology_demographics", "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"]) \
.setOutputCol("ner")
ner_converter = NerConverter() \
.setInputCols(["sentence", "token", "ner"]) \
.setOutputCol("ner_chunk")
pipeline = Pipeline(stages=[document_assembler,
sentence_detector,
tokenizer,
word_embeddings,
ner,
ner_converter])
data = spark.createDataFrame([["The patient is a 40-year-old man with history of heavy smoking."]]).toDF("text")
result = pipeline.fit(data).transform(data)
val document_assembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")
.setInputCols("document")
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner = MedicalNerModel.pretrained("ner_oncology_demographics", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")
val ner_converter = new NerConverter()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")
val pipeline = new Pipeline().setStages(Array(document_assembler,
sentence_detector,
tokenizer,
word_embeddings,
ner,
ner_converter))
val data = Seq("The patient is a 40-year-old man with history of heavy smoking.").toDS.toDF("text")
val result = pipeline.fit(data).transform(data)
import nlu
nlu.load("en.med_ner.oncology_demographics").predict("""The patient is a 40-year-old man with history of heavy smoking.""")
Results
| chunk | ner_label |
|:------------|:---------------|
| 40-year-old | Age |
| man | Gender |
| smoking | Smoking_Status |
Model Information
| Model Name: | ner_oncology_demographics |
| Compatibility: | Spark NLP for Healthcare 4.0.0+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
| Size: | 34.6 MB |
| Dependencies: | embeddings_clinical |
References
In-house annotated oncology case reports.
Benchmarking
label tp fp fn total precision recall f1
Smoking_Status 60 19 8 68 0.76 0.88 0.82
Age 934 33 15 949 0.97 0.98 0.97
Race_Ethnicity 57 5 5 62 0.92 0.92 0.92
Gender 1248 18 6 1254 0.99 1.00 0.99
macro_avg 2299 75 34 2333 0.91 0.95 0.93
micro_avg 2299 75 34 2333 0.97 0.99 0.98