Description
This model maps extracted clinical NER entities to Logical Observation Identifiers Names and Codes (LOINC) codes using bge_base_en_v1_5_onnx embeddings.
Trained on the augmented version of the LOINC 2.83 dataset (LOINC 2.83 official data plus an in-house curated dataset), scoped to numeric LOINC codes, without the inclusion of LOINC’s non-numeric Part, Answer, Panel, Survey, and other auxiliary/document codes.
If you also need LOINC’s non-numeric auxiliary codes, use bgeresolve_loinc_augmented_2_83 instead.
It also provides the official resolution of the codes within the brackets.
Live Demo Open in Colab Copy S3 URI
How to use
documentAssembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentenceDetectorDL = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")\
.setInputCols(["sentence","token"])\
.setOutputCol("embeddings")
ner_radiology = MedicalNerModel.pretrained("ner_radiology","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_radiology")
ner_converter_radiology = NerConverterInternal()\
.setInputCols(["sentence","token","ner_radiology"])\
.setOutputCol("ner_chunk_radiology")\
.setWhiteList(["Test"])
ner_jsl = MedicalNerModel.pretrained("ner_jsl","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_jsl")
ner_converter_jsl = NerConverterInternal()\
.setInputCols(["sentence","token","ner_jsl"])\
.setOutputCol("ner_chunk_jsl")\
.setWhiteList(["Test"])
chunk_merger = ChunkMergeApproach()\
.setInputCols("ner_chunk_jsl", "ner_chunk_radiology")\
.setOutputCol("ner_chunk")
chunk2doc = Chunk2Doc()\
.setInputCols(["ner_chunk"])\
.setOutputCol("ner_chunk_doc")
embedder = BGEEmbeddings.pretrained("bge_base_en_v1_5_onnx", "en")\
.setInputCols(["ner_chunk_doc"])\
.setOutputCol("bge_embeddings")
resolver = SentenceEntityResolverModel.pretrained("bgeresolve_loinc_numeric_augmented_2_83","en","clinical/models")\
.setInputCols(["bge_embeddings"])\
.setOutputCol("resolution")\
.setDistanceFunction("EUCLIDEAN")
pipeline = Pipeline(stages=[\
documentAssembler, sentenceDetectorDL, tokenizer, word_embeddings,\
ner_radiology, ner_converter_radiology, ner_jsl, ner_converter_jsl,\
chunk_merger, chunk2doc,\
embedder, resolver\
])
data = spark.createDataFrame([["The patient's glucose and hemoglobin A1c levels were checked, along with a basic metabolic panel including sodium and potassium, and a complete blood count."]]).toDF("text")
result = pipeline.fit(data).transform(data)
documentAssembler = nlp.DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentenceDetectorDL = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = nlp.Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
word_embeddings = nlp.WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")\
.setInputCols(["sentence","token"])\
.setOutputCol("embeddings")
ner_radiology = medical.NerModel.pretrained("ner_radiology","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_radiology")
ner_converter_radiology = medical.NerConverterInternal()\
.setInputCols(["sentence","token","ner_radiology"])\
.setOutputCol("ner_chunk_radiology")\
.setWhiteList(["Test"])
ner_jsl = medical.NerModel.pretrained("ner_jsl","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_jsl")
ner_converter_jsl = medical.NerConverterInternal()\
.setInputCols(["sentence","token","ner_jsl"])\
.setOutputCol("ner_chunk_jsl")\
.setWhiteList(["Test"])
chunk_merger = medical.ChunkMergeApproach()\
.setInputCols("ner_chunk_jsl", "ner_chunk_radiology")\
.setOutputCol("ner_chunk")
chunk2doc = nlp.Chunk2Doc()\
.setInputCols(["ner_chunk"])\
.setOutputCol("ner_chunk_doc")
embedder = nlp.BGEEmbeddings.pretrained("bge_base_en_v1_5_onnx", "en")\
.setInputCols(["ner_chunk_doc"])\
.setOutputCol("bge_embeddings")
resolver = medical.SentenceEntityResolverModel.pretrained("bgeresolve_loinc_numeric_augmented_2_83","en","clinical/models")\
.setInputCols(["bge_embeddings"])\
.setOutputCol("resolution")\
.setDistanceFunction("EUCLIDEAN")
pipeline = nlp.Pipeline(stages=[\
documentAssembler, sentenceDetectorDL, tokenizer, word_embeddings,\
ner_radiology, ner_converter_radiology, ner_jsl, ner_converter_jsl,\
chunk_merger, chunk2doc,\
embedder, resolver\
])
data = spark.createDataFrame([["The patient's glucose and hemoglobin A1c levels were checked, along with a basic metabolic panel including sodium and potassium, and a complete blood count."]]).toDF("text")
result = pipeline.fit(data).transform(data)
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentenceDetectorDL = SentenceDetectorDLModel
.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")
.setInputCols(Array("document"))
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val word_embeddings = WordEmbeddingsModel
.pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner_radiology = MedicalNerModel
.pretrained("ner_radiology", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner_radiology")
val ner_converter_radiology = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner_radiology"))
.setOutputCol("ner_chunk_radiology")
.setWhiteList(Array("Test"))
val ner_jsl = MedicalNerModel
.pretrained("ner_jsl", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner_jsl")
val ner_converter_jsl = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner_jsl"))
.setOutputCol("ner_chunk_jsl")
.setWhiteList(Array("Test"))
val chunk_merger = new ChunkMergeApproach()
.setInputCols(Array("ner_chunk_jsl", "ner_chunk_radiology"))
.setOutputCol("ner_chunk")
val chunk2doc = new Chunk2Doc()
.setInputCols(Array("ner_chunk"))
.setOutputCol("ner_chunk_doc")
val embedder = BGEEmbeddings
.pretrained("bge_base_en_v1_5_onnx", "en")
.setInputCols(Array("ner_chunk_doc"))
.setOutputCol("bge_embeddings")
val resolver = SentenceEntityResolverModel
.pretrained("bgeresolve_loinc_numeric_augmented_2_83", "en", "clinical/models")
.setInputCols(Array("bge_embeddings"))
.setOutputCol("resolution")
.setDistanceFunction("EUCLIDEAN")
val pipeline = new Pipeline().setStages(Array(
documentAssembler, sentenceDetectorDL, tokenizer, word_embeddings,
ner_radiology, ner_converter_radiology, ner_jsl, ner_converter_jsl,
chunk_merger, chunk2doc,
embedder, resolver
))
val data = Seq("The patient's glucose and hemoglobin A1c levels were checked, along with a basic metabolic panel including sodium and potassium, and a complete blood count.").toDF("text")
val res = pipeline.fit(data).transform(data)
Results
| ner_chunk | entity | LOINC Code | Resolution | all_k_results | all_k_cosine_distances | all_k_resolutions |
|:----------------------|:---------|:-------------|:----------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| glucose | Test | 2345-7 | glucose [Glucose [Mass/volume] in Serum or Plasma] | 2345-7:::54486-6:::104638-2:::62418-9:::50016-5:::2344-0:::6883-3:::4269-7:::234... | 0.0000:::0.1360:::0.1421:::0.1465:::0.1497:::0.1564:::0.1604:::0.1643:::0.1693::... | glucose [Glucose [Mass/volume] in Serum or Plasma]:::Glucose, Water [Glucose [Ma... |
| hemoglobin A1c levels | Test | 41995-2 | hemoglobin a1c [Hemoglobin A1c [Mass/volume] in Blood] | 41995-2:::4548-4:::86910-7:::17856-6:::112870-1:::17855-8:::71875-9:::4549-2:::9... | 0.0678:::0.1064:::0.1390:::0.1409:::0.1531:::0.1654:::0.1654:::0.1668:::0.1808::... | hemoglobin a1c [Hemoglobin A1c [Mass/volume] in Blood]:::Hgb A1c [Hemoglobin A1c... |
| basic metabolic panel | Test | 51990-0 | basic metabolic panel [Basic metabolic panel - Blood] | 51990-0:::89044-2:::101655-9:::24321-2:::24320-4:::104076-5:::54233-2:::70219-1:... | 0.0000:::0.0534:::0.0710:::0.0733:::0.0872:::0.1042:::0.1393:::0.1453:::0.1473::... | basic metabolic panel [Basic metabolic panel - Blood]:::basic metabolic & albumi... |
| sodium | Test | 2951-2 | sodium [Sodium [Moles/volume] in Serum or Plasma] | 2951-2:::9485-4:::81011-9:::28003-2:::13895-8:::12907-2:::2950-4:::23915-2:::569... | 0.0000:::0.1135:::0.1499:::0.1508:::0.1517:::0.1635:::0.1665:::0.1727:::0.1776::... | sodium [Sodium [Moles/volume] in Serum or Plasma]:::Sodium, Water [Sodium [Mass/... |
| potassium | Test | 2823-3 | potassium [Potassium [Moles/volume] in Serum or Plasma] | 2823-3:::9482-1:::10322-6:::59733-6:::2821-7:::2824-1:::2825-8:::2828-2:::32550-... | 0.0000:::0.1425:::0.1556:::0.1598:::0.1764:::0.1780:::0.1784:::0.1802:::0.1817::... | potassium [Potassium [Moles/volume] in Serum or Plasma]:::Potassium, Water [Pota... |
| complete blood count | Test | 24358-4 | Complete Blood Count [Hemogram without Platelets panel - Blood] | 24358-4:::58410-2:::47288-6:::74412-8:::11282-1:::51876-1:::1335-9:::789-8:::570... | 0.0000:::0.1182:::0.1736:::0.1781:::0.1812:::0.1821:::0.1907:::0.2021:::0.2065::... | Complete Blood Count [Hemogram without Platelets panel - Blood]:::complete blood... |
Model Information
| Model Name: | bgeresolve_loinc_numeric_augmented_2_83 |
| Compatibility: | Healthcare NLP 6.4.1+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [bge_embeddings] |
| Output Labels: | [loinc_code] |
| Language: | en |
| Size: | 944.3 MB |
| Case sensitive: | false |