Description
An NER model to extract all types of anatomical references in text using “embeddings_clinical” embeddings. It is a single entity model and generalizes all anatomical references to a single entity.
Predicted Entities
Anatomy
Live Demo Open in Colab Download
How to use
...
word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\
.setInputCols(["sentence", "token"])\
.setOutputCol("embeddings")
clinical_ner = MedicalNerModel.pretrained("ner_anatomy_coarse", "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"]) \
.setOutputCol("ner")
...
nlp_pipeline = Pipeline(stages=[document_assembler, sentence_detector, tokenizer, word_embeddings, clinical_ner, ner_converter])
model = nlpPipeline.fit(spark.createDataFrame([["content in the lung tissue"]]).toDF("text"))
results = model.transform(data)
...
val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner = MedicalNerModel.pretrained("ner_anatomy_coarse", "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"]) \
.setOutputCol("ner")
...
val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, word_embeddings, ner, ner_converter))
val result = pipeline.fit(Seq.empty["content in the lung tissue"].toDS.toDF("text")).transform(data)
Results
| | ner_chunk | entity |
|---:|------------------:|----------:|
| 0 | lung tissue | Anatomy |
Model Information
| Model Name: | ner_anatomy_coarse |
| Compatibility: | Spark NLP for Healthcare 3.0.0+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence, token, embeddings] |
| Output Labels: | [ner] |
| Language: | en |
Data Source
Trained on a custom dataset using ‘embeddings_clinical’.
Benchmarking
| | label | tp | fp | fn | prec | rec | f1 |
|---:|:--------------|------:|------:|------:|---------:|---------:|---------:|
| 0 | B-Anatomy | 2568 | 165 | 158 | 0.939627 | 0.94204 | 0.940832 |
| 1 | I-Anatomy | 1692 | 89 | 169 | 0.950028 | 0.909189 | 0.92916 |
| 2 | Macro-average | 4260 | 254 | 327 | 0.944827 | 0.925614 | 0.935122 |
| 3 | Micro-average | 4260 | 254 | 327 | 0.943731 | 0.928712 | 0.936161 |