Description
Detect adverse reactions of drugs in reviews, tweets, and medical text using pretrained NER model.
Predicted Entities
DRUG, ADE
Live Demo Open in Colab Download
How to use
...
embeddings_clinical = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models") .setInputCols(["sentence", "token"]) .setOutputCol("embeddings")
clinical_ner = MedicalNerModel.pretrained("ner_ade_clinical", "en", "clinical/models") .setInputCols(["sentence", "token", "embeddings"]) .setOutputCol("ner")
...
nlpPipeline = Pipeline(stages=[document_assembler, sentence_detector, tokenizer, embeddings_clinical, clinical_ner, ner_converter])
model = nlpPipeline.fit(spark.createDataFrame([[""]]).toDF("text"))
results = model.transform(spark.createDataFrame([["EXAMPLE_TEXT"]]).toDF("text"))
...
val embeddings_clinical = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner = MedicalNerModel.pretrained("ner_ade_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")
...
val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, embeddings_clinical, ner, ner_converter))
val result = pipeline.fit(Seq.empty[String]).transform(data)
Model Information
| Model Name: | ner_ade_clinical |
| Compatibility: | Spark NLP for Healthcare 3.0.0+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
Benchmarking
+------+-------+------+------+-------+---------+------+------+
|entity| tp| fp| fn| total|precision|recall| f1|
+------+-------+------+------+-------+---------+------+------+
| DRUG|17470.0|1436.0|1951.0|19421.0| 0.924|0.8995|0.9116|
| ADE| 6010.0|1244.0|1886.0| 7896.0| 0.8285|0.7611|0.7934|
+------+-------+------+------+-------+---------+------+------+
+------------------+
| macro|
+------------------+
|0.8525141088742945|
+------------------+
+------------------+
| micro|
+------------------+
|0.8774545383517981|
+------------------+