Description
This model extracts healthcare-related terms from the documents transferred from the patient’s own sentences.
Predicted Entities
Gender, Employment, BodyPart, Vaccine, Age, PsychologicalCondition, Form, Drug, Substance, ClinicalDept, Laterality, DateTime, Test, Route, Disease, AdmissionDischarge, Dosage, Duration, VitalTest, Frequency, Symptom, Allergen, Procedure, RelationshipStatus, HealthStatus, Modifier, TestResult, InjuryOrPoisoning, SubstanceQuantity, MedicalDevice, Treatment
Live Demo Open in Colab Copy S3 URI
How to use
document_assembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer() \
.setInputCols(["sentence"]) \
.setOutputCol("token")
word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical_medium", "en", "clinical/models")\
.setInputCols(["sentence", "token"]) \
.setOutputCol("embeddings")
ner = MedicalNerModel.pretrained("ner_vop_emb_clinical_medium", "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"]) \
.setOutputCol("ner")
ner_converter = NerConverterInternal() \
.setInputCols(["sentence", "token", "ner"]) \
.setOutputCol("ner_chunk")
pipeline = Pipeline(stages=[document_assembler,
sentence_detector,
tokenizer,
word_embeddings,
ner,
ner_converter])
data = spark.createDataFrame([["Hello,I'm 20 year old girl. I'm diagnosed with hyperthyroid 1 month ago. I was feeling weak, light headed,poor digestion, panic attacks, depression, left chest pain, increased heart rate, rapidly weight loss, from 4 months. Because of this, I stayed in the hospital and just discharged from hospital. I had many other blood tests, brain mri, ultrasound scan, endoscopy because of some dumb doctors bcs they were not able to diagnose actual problem. Finally I got an appointment with a homeopathy doctor finally he find that i was suffering from hyperthyroid and my TSH was 0.15 T3 and T4 is normal . Also i have b12 deficiency and vitamin D deficiency so I'm taking weekly supplement of vitamin D and 1000 mcg b12 daily. I'm taking homeopathy medicine for 40 days and took 2nd test after 30 days. My TSH is 0.5 now. I feel a little bit relief from weakness and depression but I'm facing with 2 new problem from last week that is breathtaking problem and very rapid heartrate. I just want to know if i should start allopathy medicine or homeopathy is okay? Bcs i heard that thyroid take time to start recover. So please let me know if both of medicines take same time. Because some of my friends advising me to start allopathy and never take a chance as i can develop some serious problems.Sorry for my poor english😐Thank you."]]).toDF("text")
result = pipeline.fit(data).transform(data)
val document_assembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")
.setInputCols("document")
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val word_embeddings = WordEmbeddingsModel().pretrained("embeddings_clinical_medium", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner = MedicalNerModel.pretrained("ner_vop_emb_clinical_medium", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")
val ner_converter = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")
val pipeline = new Pipeline().setStages(Array(document_assembler,
sentence_detector,
tokenizer,
word_embeddings,
ner,
ner_converter))
val data = Seq("Hello,I'm 20 year old girl. I'm diagnosed with hyperthyroid 1 month ago. I was feeling weak, light headed,poor digestion, panic attacks, depression, left chest pain, increased heart rate, rapidly weight loss, from 4 months. Because of this, I stayed in the hospital and just discharged from hospital. I had many other blood tests, brain mri, ultrasound scan, endoscopy because of some dumb doctors bcs they were not able to diagnose actual problem. Finally I got an appointment with a homeopathy doctor finally he find that i was suffering from hyperthyroid and my TSH was 0.15 T3 and T4 is normal . Also i have b12 deficiency and vitamin D deficiency so I'm taking weekly supplement of vitamin D and 1000 mcg b12 daily. I'm taking homeopathy medicine for 40 days and took 2nd test after 30 days. My TSH is 0.5 now. I feel a little bit relief from weakness and depression but I'm facing with 2 new problem from last week that is breathtaking problem and very rapid heartrate. I just want to know if i should start allopathy medicine or homeopathy is okay? Bcs i heard that thyroid take time to start recover. So please let me know if both of medicines take same time. Because some of my friends advising me to start allopathy and never take a chance as i can develop some serious problems.Sorry for my poor english😐Thank you.").toDS.toDF("text")
val result = pipeline.fit(data).transform(data)
Results
| chunk | ner_label |
|:---------------------|:-----------------------|
| 20 year old | Age |
| girl | Gender |
| hyperthyroid | Disease |
| 1 month ago | DateTime |
| weak | Symptom |
| light | Symptom |
| panic attacks | PsychologicalCondition |
| depression | PsychologicalCondition |
| left | Laterality |
| chest | BodyPart |
| pain | Symptom |
| increased | TestResult |
| heart rate | VitalTest |
| rapidly | Modifier |
| weight loss | Symptom |
| 4 months | Duration |
| hospital | ClinicalDept |
| discharged | AdmissionDischarge |
| hospital | ClinicalDept |
| blood tests | Test |
| brain | BodyPart |
| mri | Test |
| ultrasound scan | Test |
| endoscopy | Procedure |
| doctors | Employment |
| homeopathy doctor | Employment |
| he | Gender |
| hyperthyroid | Disease |
| TSH | Test |
| 0.15 | TestResult |
| T3 | Test |
| T4 | Test |
| normal | TestResult |
| b12 deficiency | Disease |
| vitamin D deficiency | Disease |
| weekly | Frequency |
| supplement | Drug |
| vitamin D | Drug |
| 1000 mcg | Dosage |
| b12 | Drug |
| daily | Frequency |
| homeopathy medicine | Drug |
| 40 days | Duration |
| after 30 days | DateTime |
| TSH | Test |
| 0.5 | TestResult |
| now | DateTime |
| weakness | Symptom |
| depression | PsychologicalCondition |
| last week | DateTime |
| rapid | TestResult |
| heartrate | VitalTest |
| homeopathy | Treatment |
| thyroid | BodyPart |
| allopathy | Drug |
Model Information
| Model Name: | ner_vop_emb_clinical_medium |
| Compatibility: | Healthcare NLP 4.4.3+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
| Size: | 3.9 MB |
| Dependencies: | embeddings_clinical_medium |
References
In-house annotated health-related text in colloquial language.
Sample text from the training dataset
Hello,I’m 20 year old girl. I’m diagnosed with hyperthyroid 1 month ago. I was feeling weak, light headed,poor digestion, panic attacks, depression, left chest pain, increased heart rate, rapidly weight loss, from 4 months. Because of this, I stayed in the hospital and just discharged from hospital. I had many other blood tests, brain mri, ultrasound scan, endoscopy because of some dumb doctors bcs they were not able to diagnose actual problem. Finally I got an appointment with a homeopathy doctor finally he find that i was suffering from hyperthyroid and my TSH was 0.15 T3 and T4 is normal . Also i have b12 deficiency and vitamin D deficiency so I’m taking weekly supplement of vitamin D and 1000 mcg b12 daily. I’m taking homeopathy medicine for 40 days and took 2nd test after 30 days. My TSH is 0.5 now. I feel a little bit relief from weakness and depression but I’m facing with 2 new problem from last week that is breathtaking problem and very rapid heartrate. I just want to know if i should start allopathy medicine or homeopathy is okay? Bcs i heard that thyroid take time to start recover. So please let me know if both of medicines take same time. Because some of my friends advising me to start allopathy and never take a chance as i can develop some serious problems.Sorry for my poor english😐Thank you.
Benchmarking
label tp fp fn total precision recall f1
Gender 1293 24 24 1317 0.98 0.98 0.98
Employment 1182 48 61 1243 0.96 0.95 0.96
BodyPart 2697 228 203 2900 0.92 0.93 0.93
Vaccine 39 4 3 42 0.91 0.93 0.92
Age 552 61 30 582 0.90 0.95 0.92
PsychologicalCondition 405 33 39 444 0.92 0.91 0.92
Form 248 30 18 266 0.89 0.93 0.91
Drug 1327 158 113 1440 0.89 0.92 0.91
Substance 387 55 34 421 0.88 0.92 0.90
ClinicalDept 293 42 33 326 0.87 0.90 0.89
Laterality 550 55 78 628 0.91 0.88 0.89
DateTime 4030 621 372 4402 0.87 0.92 0.89
Test 1056 134 152 1208 0.89 0.87 0.88
Route 42 7 6 48 0.86 0.88 0.87
Disease 1750 312 265 2015 0.85 0.87 0.86
AdmissionDischarge 27 2 7 34 0.93 0.79 0.86
Dosage 346 57 66 412 0.86 0.84 0.85
Duration 1889 287 421 2310 0.87 0.82 0.84
VitalTest 144 28 28 172 0.84 0.84 0.84
Frequency 889 171 190 1079 0.84 0.82 0.83
Symptom 3707 671 868 4575 0.85 0.81 0.83
Allergen 34 2 12 46 0.94 0.74 0.83
Procedure 583 140 122 705 0.81 0.83 0.82
RelationshipStatus 18 2 6 24 0.90 0.75 0.82
HealthStatus 83 29 24 107 0.74 0.78 0.76
Modifier 787 183 352 1139 0.81 0.69 0.75
TestResult 362 112 162 524 0.76 0.69 0.73
InjuryOrPoisoning 119 35 57 176 0.77 0.68 0.72
SubstanceQuantity 59 20 26 85 0.75 0.69 0.72
MedicalDevice 228 92 104 332 0.71 0.69 0.70
Treatment 144 45 84 228 0.76 0.63 0.69
macro_avg 25270 3688 3960 29230 0.86 0.83 0.85
micro_avg 25270 3688 3960 29230 0.87 0.87 0.87