Description
Detect adverse reactions to drugs in reviews, tweets, and medical text using a pre-trained NER model.
Predicted Entities
DRUG, ADE
Live Demo Open in Colab Copy S3 URI
How to use
document_assembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical_medium", "en", "clinical/models")\
.setInputCols(["sentence", "token"])\
.setOutputCol("embeddings")
ner_model = MedicalNerModel.pretrained("ner_ade_emb_clinical_medium", "en", "clinical/models")\
.setInputCols(["sentence", "token", "embeddings"])\
.setOutputCol("ner")
ner_converter = NerConverterInternal()\
.setInputCols(['sentence', 'token', 'ner'])\
.setOutputCol('ner_chunk')
pipeline = Pipeline(stages=[
document_assembler,
sentence_detector,
tokenizer,
clinical_embeddings,
ner_model,
ner_converter
])
sample_df = spark.createDataFrame([["Been taking Lipitor for 15 years , have experienced severe fatigue a lot!!! . Doctor moved me to voltaren 2 months ago , so far , have only experienced cramps."]]).toDF("text")
result = pipeline.fit(sample_df).transform(sample_df)
val document_assembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")
.setInputCols("document")
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical_medium", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner_model = MedicalNerModel.pretrained("ner_ade_emb_clinical_medium", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")
val ner_converter = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")
val pipeline = new Pipeline().setStages(Array(
document_assembler,
sentence_detector,
tokenizer,
clinical_embeddings,
ner_model,
ner_converter))
val sample_data = Seq("Been taking Lipitor for 15 years , have experienced severe fatigue a lot!!! . Doctor moved me to voltaren 2 months ago , so far , have only experienced cramps.").toDS.toDF("text")
val result = pipeline.fit(sample_data).transform(sample_data)
Results
+--------------+-----+---+---------+
|chunk |begin|end|ner_label|
+--------------+-----+---+---------+
|Lipitor |12 |18 |DRUG |
|severe fatigue|52 |65 |ADE |
|voltaren |97 |104|DRUG |
|cramps |152 |157|ADE |
+--------------+-----+---+---------+
Model Information
| Model Name: | ner_ade_emb_clinical_medium |
| Compatibility: | Healthcare NLP 4.4.2+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
| Size: | 2.7 MB |
Benchmarking
label precision recall f1-score support
DRUG 0.92 0.91 0.91 15895
ADE 0.83 0.77 0.80 6077
micro-avg 0.89 0.87 0.88 21972
macro-avg 0.87 0.84 0.86 21972
weighted-avg 0.89 0.87 0.88 21972