Detect Adverse Drug Events

Description

Detect adverse reactions of drugs in reviews, tweets, and medical text using pretrained NER model.

Predicted Entities

DRUG, ADE

Live Demo Open in Colab Copy S3 URI

How to use

document_assembler = DocumentAssembler()\
    .setInputCol("text")\
    .setOutputCol("document")

sentence_detector = SentenceDetector()\
    .setInputCols(["document"])\
    .setOutputCol("sentence")

tokenizer = Tokenizer()\
    .setInputCols(["sentence"])\
    .setOutputCol("token")

embeddings_clinical = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\
    .setInputCols(["sentence", "token"])\
    .setOutputCol("embeddings")

clinical_ner = MedicalNerModel.pretrained("ner_ade_clinical", "en", "clinical/models")\
    .setInputCols(["sentence", "token", "embeddings"])\
    .setOutputCol("ner")

ner_converter = NerConverter()\
 	  .setInputCols(["sentence", "token", "ner"])\
 	  .setOutputCol("ner_chunk")

nlpPipeline = Pipeline(stages=[document_assembler,
                               sentence_detector,
                               tokenizer,
                               embeddings_clinical,
                               clinical_ner,
                               ner_converter])

model = nlpPipeline.fit(spark.createDataFrame([[""]]).toDF("text"))

result = model.transform(spark.createDataFrame([["Hypersensitivity to aspirin can be manifested as acute asthma, urticaria and/or angioedema, or a systemic anaphylactoid reaction."]]).toDF("text"))
val document_assembler = new DocumentAssembler()
	.setInputCol("text")
	.setOutputCol("document")
	
val sentence_detector = new SentenceDetector()
	.setInputCols(Array("document"))
	.setOutputCol("sentence")
	
val tokenizer = new Tokenizer()
	.setInputCols(Array("sentence"))
	.setOutputCol("token")
	
val embeddings_clinical = WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")
	.setInputCols(Array("sentence","token"))
	.setOutputCol("embeddings")
	
val clinical_ner = MedicalNerModel.pretrained("ner_ade_clinical","en","clinical/models")
	.setInputCols(Array("sentence","token","embeddings"))
	.setOutputCol("ner")
	
val ner_converter = new NerConverter()
	.setInputCols(Array("sentence","token","ner"))
	.setOutputCol("ner_chunk")
	
val nlpPipeline = new Pipeline().setStages(Array(
		 document_assembler, 
		 sentence_detector, 
		 tokenizer, 
		 embeddings_clinical, 
		 clinical_ner, 
		 ner_converter))
	
val model = nlpPipeline.fit(Seq("").toDF("text"))
	
val result = model.transform(Seq("Hypersensitivity to aspirin can be manifested as acute asthma, urticaria and/or angioedema, or a systemic anaphylactoid reaction.").toDF("text"))
import nlu
nlu.load("en.med_ner.ade.clinical").predict("""Hypersensitivity to aspirin can be manifested as acute asthma, urticaria and/or angioedema, or a systemic anaphylactoid reaction.""")

Results

+-------------------------------+-----+---+---------+
|                          chunk|begin|end|ner_label|
+-------------------------------+-----+---+---------+
|                        aspirin|   20| 26|     DRUG|
|                   acute asthma|   49| 60|      ADE|
|                      urticaria|   63| 71|      ADE|
|                     angioedema|   80| 89|      ADE|
|systemic anaphylactoid reaction|   97|127|      ADE|
+-------------------------------+-----+---+---------+

Model Information

Model Name: ner_ade_clinical
Compatibility: Healthcare NLP 3.0.0+
License: Licensed
Edition: Official
Input Labels: [sentence, token, embeddings]
Output Labels: [ner]
Language: en

Benchmarking

+------+-------+------+------+-------+---------+------+------+
|entity|     tp|    fp|    fn|  total|precision|recall|    f1|
+------+-------+------+------+-------+---------+------+------+
|  DRUG|17470.0|1436.0|1951.0|19421.0|    0.924|0.8995|0.9116|
|   ADE| 6010.0|1244.0|1886.0| 7896.0|   0.8285|0.7611|0.7934|
+------+-------+------+------+-------+---------+------+------+

+------------------+
|             macro|
+------------------+
|0.8525141088742945|
+------------------+

+------------------+
|             micro|
+------------------+
|0.8774545383517981|
+------------------+