Description
This model extracts demographic information from oncology texts, including age, gender and smoking status.
Definitions of Predicted Entities:
Age: All mention of ages, past or present, related to the patient or with anybody else.Gender: Gender-specific nouns and pronouns (including words such as “him” or “she”, and family members such as “father”).Race_Ethnicity: The race and ethnicity categories include racial and national origin or sociocultural groups.Smoking_Status: All mentions of smoking related to the patient or to someone else.
Predicted Entities
Age, Gender, Race_Ethnicity, Smoking_Status
Live Demo Open in Colab Copy S3 URI
How to use
document_assembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer() \
.setInputCols(["sentence"]) \
.setOutputCol("token")\
.setSplitChars(['-'])
word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical", "en", "clinical/models")\
.setInputCols(["sentence", "token"]) \
.setOutputCol("embeddings")
ner = MedicalNerModel.pretrained("ner_oncology_demographics_wip", "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"]) \
.setOutputCol("ner")
ner_converter = NerConverter() \
.setInputCols(["sentence", "token", "ner"]) \
.setOutputCol("ner_chunk")
pipeline = Pipeline(stages=[document_assembler,
sentence_detector,
tokenizer,
word_embeddings,
ner,
ner_converter])
data = spark.createDataFrame([["The patient is a 40-year-old man with history of heavy smoking."]]).toDF("text")
result = pipeline.fit(data).transform(data)
val document_assembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")
.setInputCols(Array("document"))
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols(Array("sentence"))
.setOutputCol("token")
.setSplitChars("-")
val word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner = MedicalNerModel.pretrained("ner_oncology_demographics_wip", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")
val ner_converter = new NerConverter()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")
val pipeline = new Pipeline().setStages(Array(document_assembler,
sentence_detector,
tokenizer,
word_embeddings,
ner,
ner_converter))
val data = Seq("The patient is a 40-year-old man with history of heavy smoking.").toDS.toDF("text")
val result = pipeline.fit(data).transform(data)
import nlu
nlu.load("en.med_ner.oncology_demographics_wip").predict("""The patient is a 40-year-old man with history of heavy smoking.""")
Results
| chunk | ner_label |
|:------------|:---------------|
| 40-year-old | Age |
| man | Gender |
| smoking | Smoking_Status |
Model Information
| Model Name: | ner_oncology_demographics_wip |
| Compatibility: | Healthcare NLP 4.0.0+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
| Size: | 849.2 KB |
References
In-house annotated oncology case reports.
Benchmarking
label tp fp fn total precision recall f1
Smoking_Status 43.0 12.0 11.0 54.0 0.78 0.80 0.79
Age 679.0 27.0 17.0 696.0 0.96 0.98 0.97
Race_Ethnicity 44.0 7.0 7.0 51.0 0.86 0.86 0.86
Gender 933.0 14.0 8.0 941.0 0.99 0.99 0.99
macro_avg 1699.0 60.0 43.0 1742.0 0.90 0.91 0.90
micro_avg NaN NaN NaN NaN 0.97 0.98 0.97