Description
An NER model to extract all types of anatomical references in text using “embeddings_clinical” embeddings. It is a single entity model and generalizes all anatomical references to a single entity.
Predicted Entities
Anatomy
Live Demo Open in Colab Copy S3 URI
How to use
document_assembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentence_detector = SentenceDetector()\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\
.setInputCols(["sentence", "token"])\
.setOutputCol("embeddings")
clinical_ner = MedicalNerModel.pretrained("ner_anatomy_coarse", "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"]) \
.setOutputCol("ner")
ner_converter = NerConverter()\
.setInputCols(["sentence", "token", "ner"])\
.setOutputCol("ner_chunk")
nlp_pipeline = Pipeline(stages=[document_assembler, sentence_detector, tokenizer, word_embeddings, clinical_ner, ner_converter])
model = nlp_pipeline.fit(spark.createDataFrame([[""]]).toDF("text"))
results = model.transform(spark.createDataFrame([["content in the lung tissue"]]).toDF("text"))
val document_assembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentence_detector = new SentenceDetector()
.setInputCols("document")
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner = MedicalNerModel.pretrained("ner_anatomy_coarse", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")
val ner_converter = new NerConverter()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")
val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, word_embeddings, ner, ner_converter))
val data = Seq("""content in the lung tissue""").toDS().toDF("text")
val result = pipeline.fit(data).transform(data)
import nlu
nlu.load("en.med_ner.anatomy.coarse").predict("""content in the lung tissue""")
Results
| | ner_chunk | entity |
|---:|------------------:|----------:|
| 0 | lung tissue | Anatomy |
Model Information
| Model Name: | ner_anatomy_coarse |
| Compatibility: | Healthcare NLP 3.0.0+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
Data Source
Trained on a custom dataset using ‘embeddings_clinical’.
Benchmarking
| | label | tp | fp | fn | prec | rec | f1 |
|---:|:--------------|------:|------:|------:|---------:|---------:|---------:|
| 0 | B-Anatomy | 2568 | 165 | 158 | 0.939627 | 0.94204 | 0.940832 |
| 1 | I-Anatomy | 1692 | 89 | 169 | 0.950028 | 0.909189 | 0.92916 |
| 2 | Macro-average | 4260 | 254 | 327 | 0.944827 | 0.925614 | 0.935122 |
| 3 | Micro-average | 4260 | 254 | 327 | 0.943731 | 0.928712 | 0.936161 |