Description
This model extracts healthcare-related terms from the documents transferred from the patient’s own sentences.
Predicted Entities
AdmissionDischarge, Age, BodyPart, ClinicalDept, DateTime, Disease, Dosage, Drug, Duration, Form, Frequency, Gender, HealthStatus, InjuryOrPoisoning, Laterality, MedicalDevice, Modifier, ProcedurePsychologicalCondition, RelationshipStatus, Route, Substance, SubstanceQuantity, Symptom, Test, TestResult, Treatment, Vaccine, VitalTest
Live Demo Open in Colab Copy S3 URI
How to use
document_assembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\
.setInputCols(["sentence", "token"])\
.setOutputCol("embeddings")
ner_model = MedicalNerModel.pretrained("ner_vop_v2", "en", "clinical/models")\
.setInputCols(["sentence", "token", "embeddings"])\
.setOutputCol("ner")
ner_converter = NerConverterInternal()\
.setInputCols(["sentence", "token", "ner"])\
.setOutputCol("ner_chunk")
pipeline = Pipeline(stages=[
document_assembler,
sentence_detector,
tokenizer,
clinical_embeddings,
ner_model,
ner_converter
])
sample_texts = ["""Hello,I'm 20 year old girl. I'm diagnosed with hyperthyroid 1 month ago. I was feeling weak, light headed,poor digestion, panic attacks, depression, left chest pain, increased heart rate, rapidly weight loss, from 4 months. Because of this, I stayed in the hospital and just discharged from hospital. I had many other blood tests, brain mri, ultrasound scan, endoscopy because of some dumb doctors bcs they were not able to diagnose actual problem. Finally I got an appointment with a homeopathy doctor finally he find that i was suffering from hyperthyroid and my TSH was 0.15 T3 and T4 is normal . Also i have b12 deficiency and vitamin D deficiency so I'm taking weekly supplement of vitamin D and 1000 mcg b12 daily. I'm taking homeopathy medicine for 40 days and took 2nd test after 30 days. My TSH is 0.5 now. I feel a little bit relief from weakness and depression but I'm facing with 2 new problem from last week that is breathtaking problem and very rapid heartrate. I just want to know if i should start allopathy medicine or homeopathy is okay? Bcs i heard that thyroid take time to start recover. So please let me know if both of medicines take same time. Because some of my friends advising me to start allopathy and never take a chance as i can develop some serious problems.Sorry for my poor english😐Thank you."""]
data = spark.createDataFrame(sample_texts, F.StringType()).toDF("text")
result = pipeline.fit(data).transform(data)
val document_assembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")
.setInputCols("document")
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner_model = MedicalNerModel.pretrained("ner_vop_v2", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")
val ner_converter = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")
val pipeline = new Pipeline().setStages(Array(
document_assembler,
sentenceDetector,
tokenizer,
clinical_embeddings,
ner_model,
ner_converter
))
val sample_texts = Seq("""Hello,I'm 20 year old girl. I'm diagnosed with hyperthyroid 1 month ago. I was feeling weak, light headed,poor digestion, panic attacks, depression, left chest pain, increased heart rate, rapidly weight loss, from 4 months. Because of this, I stayed in the hospital and just discharged from hospital. I had many other blood tests, brain mri, ultrasound scan, endoscopy because of some dumb doctors bcs they were not able to diagnose actual problem. Finally I got an appointment with a homeopathy doctor finally he find that i was suffering from hyperthyroid and my TSH was 0.15 T3 and T4 is normal . Also i have b12 deficiency and vitamin D deficiency so I'm taking weekly supplement of vitamin D and 1000 mcg b12 daily. I'm taking homeopathy medicine for 40 days and took 2nd test after 30 days. My TSH is 0.5 now. I feel a little bit relief from weakness and depression but I'm facing with 2 new problem from last week that is breathtaking problem and very rapid heartrate. I just want to know if i should start allopathy medicine or homeopathy is okay? Bcs i heard that thyroid take time to start recover. So please let me know if both of medicines take same time. Because some of my friends advising me to start allopathy and never take a chance as i can develop some serious problems.Sorry for my poor english😐Thank you.""").toDF("text")
val result = pipeline.fit(sample_texts).transform(sample_texts)
Results
+--------------------+-----+----+----------------------+
|chunk |begin|end |ner_label |
+--------------------+-----+----+----------------------+
|20 year old |15 |25 |Age |
|girl |27 |30 |Gender |
|hyperthyroid |52 |63 |Disease |
|1 month ago |65 |75 |DateTime |
|weak |92 |95 |Symptom |
|light |98 |102 |Symptom |
|panic attacks |127 |139 |PsychologicalCondition|
|depression |142 |151 |PsychologicalCondition|
|left |154 |157 |Laterality |
|chest |159 |163 |BodyPart |
|pain |165 |168 |Symptom |
|increased |171 |179 |TestResult |
|heart rate |181 |190 |VitalTest |
|rapidly |193 |199 |Modifier |
|weight loss |201 |211 |Symptom |
|4 months |220 |227 |Duration |
|hospital |263 |270 |ClinicalDept |
|discharged |281 |290 |AdmissionDischarge |
|hospital |297 |304 |ClinicalDept |
|blood tests |324 |334 |Test |
|brain |337 |341 |BodyPart |
|mri |343 |345 |Test |
|ultrasound scan |348 |362 |Test |
|endoscopy |365 |373 |Procedure |
|he |517 |518 |Gender |
|hyperthyroid |551 |562 |Disease |
|TSH |571 |573 |Test |
|0.15 |579 |582 |TestResult |
|T3 |584 |585 |Test |
|T4 |591 |592 |Test |
|normal |597 |602 |TestResult |
|b12 deficiency |618 |631 |Disease |
|vitamin D deficiency|637 |656 |Disease |
|weekly |672 |677 |Frequency |
|supplement |679 |688 |Drug |
|vitamin D |693 |701 |Drug |
|1000 mcg |707 |714 |Dosage |
|b12 |716 |718 |Drug |
|daily |720 |724 |Frequency |
|homeopathy medicine |738 |756 |Drug |
|40 days |762 |768 |Duration |
|after 30 days |788 |800 |DateTime |
|TSH |806 |808 |Test |
|0.5 |813 |815 |TestResult |
|now |817 |819 |DateTime |
|weakness |854 |861 |Symptom |
|depression |867 |876 |PsychologicalCondition|
|last week |917 |925 |DateTime |
|rapid |965 |969 |TestResult |
|heartrate |971 |979 |VitalTest |
|allopathy medicine |1020 |1037|Drug |
|homeopathy |1042 |1051|Treatment |
|thyroid |1079 |1085|BodyPart |
+--------------------+-----+----+----------------------+
Model Information
| Model Name: | ner_vop_v2 |
| Compatibility: | Healthcare NLP 5.3.3+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
| Size: | 2.1 MB |
Benchmarking
label precision recall f1-score support
AdmissionDischarge 0.96 0.87 0.92 31
Age 0.83 0.83 0.83 339
BodyPart 0.89 0.91 0.90 2704
ClinicalDept 0.85 0.80 0.82 220
DateTime 0.81 0.80 0.80 2132
Disease 0.83 0.79 0.81 1535
Dosage 0.82 0.70 0.75 273
Drug 0.89 0.86 0.88 1201
Duration 0.77 0.73 0.75 1052
Form 0.87 0.90 0.88 239
Frequency 0.73 0.78 0.75 624
Gender 0.99 0.98 0.99 1282
HealthStatus 0.78 0.72 0.75 101
InjuryOrPoisoning 0.73 0.53 0.62 158
Laterality 0.84 0.83 0.83 534
MedicalDevice 0.62 0.56 0.59 234
Modifier 0.82 0.68 0.74 1064
Procedure 0.82 0.68 0.74 360
PsychologicalCondition 0.93 0.89 0.91 419
RelationshipStatus 1.00 0.74 0.85 19
Route 0.79 0.64 0.71 36
Substance 0.82 0.86 0.84 404
SubstanceQuantity 0.71 0.56 0.63 54
Symptom 0.83 0.81 0.82 3517
Test 0.80 0.81 0.81 755
TestResult 0.72 0.61 0.66 534
Treatment 0.71 0.25 0.37 155
Vaccine 0.88 0.54 0.67 28
VitalTest 0.88 0.73 0.80 144
micro-avg 0.84 0.81 0.82 20148
macro-avg 0.83 0.74 0.77 20148
weighted-avg 0.84 0.81 0.82 20148