Description
This model maps extracted clinical NER entities to Logical Observation Identifiers Names and Codes (LOINC) codes using mpnet_embeddings_biolord_2023_c embeddings.
Trained on the official LOINC 2.83 dataset.
This model may also resolve to non-numeric LOINC codes, including LOINC’s own internal Part codes (prefixed “LP”). If you only need numeric, directly orderable/reportable LOINC codes, use biolordresolve_loinc_numeric_2_83 instead.
It also provides the official resolution of the codes within the brackets.
Live Demo Open in Colab Copy S3 URI
How to use
documentAssembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentenceDetectorDL = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")\
.setInputCols(["sentence","token"])\
.setOutputCol("embeddings")
ner_radiology = MedicalNerModel.pretrained("ner_radiology","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_radiology")
ner_converter_radiology = NerConverterInternal()\
.setInputCols(["sentence","token","ner_radiology"])\
.setOutputCol("ner_chunk_radiology")\
.setWhiteList(["Test"])
ner_jsl = MedicalNerModel.pretrained("ner_jsl","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_jsl")
ner_converter_jsl = NerConverterInternal()\
.setInputCols(["sentence","token","ner_jsl"])\
.setOutputCol("ner_chunk_jsl")\
.setWhiteList(["Test"])
chunk_merger = ChunkMergeApproach()\
.setInputCols("ner_chunk_jsl", "ner_chunk_radiology")\
.setOutputCol("ner_chunk")
chunk2doc = Chunk2Doc()\
.setInputCols(["ner_chunk"])\
.setOutputCol("ner_chunk_doc")
embedder = MPNetEmbeddings.pretrained("mpnet_embeddings_biolord_2023_c", "en")\
.setInputCols(["ner_chunk_doc"])\
.setOutputCol("biolord_embeddings")\
.setBatchSize(1)
resolver = SentenceEntityResolverModel.pretrained("biolordresolve_loinc_2_83","en","clinical/models")\
.setInputCols(["biolord_embeddings"])\
.setOutputCol("resolution")\
.setDistanceFunction("EUCLIDEAN")
pipeline = Pipeline(stages=[\
documentAssembler, sentenceDetectorDL, tokenizer, word_embeddings,\
ner_radiology, ner_converter_radiology, ner_jsl, ner_converter_jsl,\
chunk_merger, chunk2doc,\
embedder, resolver\
])
data = spark.createDataFrame([["The patient's glucose and hemoglobin A1c levels were checked, along with a basic metabolic panel including sodium and potassium, and a complete blood count."]]).toDF("text")
result = pipeline.fit(data).transform(data)
documentAssembler = nlp.DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentenceDetectorDL = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = nlp.Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
word_embeddings = nlp.WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")\
.setInputCols(["sentence","token"])\
.setOutputCol("embeddings")
ner_radiology = medical.NerModel.pretrained("ner_radiology","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_radiology")
ner_converter_radiology = medical.NerConverterInternal()\
.setInputCols(["sentence","token","ner_radiology"])\
.setOutputCol("ner_chunk_radiology")\
.setWhiteList(["Test"])
ner_jsl = medical.NerModel.pretrained("ner_jsl","en","clinical/models")\
.setInputCols(["sentence","token","embeddings"])\
.setOutputCol("ner_jsl")
ner_converter_jsl = medical.NerConverterInternal()\
.setInputCols(["sentence","token","ner_jsl"])\
.setOutputCol("ner_chunk_jsl")\
.setWhiteList(["Test"])
chunk_merger = medical.ChunkMergeApproach()\
.setInputCols("ner_chunk_jsl", "ner_chunk_radiology")\
.setOutputCol("ner_chunk")
chunk2doc = nlp.Chunk2Doc()\
.setInputCols(["ner_chunk"])\
.setOutputCol("ner_chunk_doc")
embedder = nlp.MPNetEmbeddings.pretrained("mpnet_embeddings_biolord_2023_c", "en")\
.setInputCols(["ner_chunk_doc"])\
.setOutputCol("biolord_embeddings")\
.setBatchSize(1)
resolver = medical.SentenceEntityResolverModel.pretrained("biolordresolve_loinc_2_83","en","clinical/models")\
.setInputCols(["biolord_embeddings"])\
.setOutputCol("resolution")\
.setDistanceFunction("EUCLIDEAN")
pipeline = nlp.Pipeline(stages=[\
documentAssembler, sentenceDetectorDL, tokenizer, word_embeddings,\
ner_radiology, ner_converter_radiology, ner_jsl, ner_converter_jsl,\
chunk_merger, chunk2doc,\
embedder, resolver\
])
data = spark.createDataFrame([["The patient's glucose and hemoglobin A1c levels were checked, along with a basic metabolic panel including sodium and potassium, and a complete blood count."]]).toDF("text")
result = pipeline.fit(data).transform(data)
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentenceDetectorDL = SentenceDetectorDLModel
.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")
.setInputCols(Array("document"))
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val word_embeddings = WordEmbeddingsModel
.pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")
val ner_radiology = MedicalNerModel
.pretrained("ner_radiology", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner_radiology")
val ner_converter_radiology = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner_radiology"))
.setOutputCol("ner_chunk_radiology")
.setWhiteList(Array("Test"))
val ner_jsl = MedicalNerModel
.pretrained("ner_jsl", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner_jsl")
val ner_converter_jsl = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner_jsl"))
.setOutputCol("ner_chunk_jsl")
.setWhiteList(Array("Test"))
val chunk_merger = new ChunkMergeApproach()
.setInputCols(Array("ner_chunk_jsl", "ner_chunk_radiology"))
.setOutputCol("ner_chunk")
val chunk2doc = new Chunk2Doc()
.setInputCols(Array("ner_chunk"))
.setOutputCol("ner_chunk_doc")
val embedder = MPNetEmbeddings
.pretrained("mpnet_embeddings_biolord_2023_c", "en")
.setInputCols(Array("ner_chunk_doc"))
.setOutputCol("biolord_embeddings")
.setBatchSize(1)
val resolver = SentenceEntityResolverModel
.pretrained("biolordresolve_loinc_2_83", "en", "clinical/models")
.setInputCols(Array("biolord_embeddings"))
.setOutputCol("resolution")
.setDistanceFunction("EUCLIDEAN")
val pipeline = new Pipeline().setStages(Array(
documentAssembler, sentenceDetectorDL, tokenizer, word_embeddings,
ner_radiology, ner_converter_radiology, ner_jsl, ner_converter_jsl,
chunk_merger, chunk2doc,
embedder, resolver
))
val data = Seq("The patient's glucose and hemoglobin A1c levels were checked, along with a basic metabolic panel including sodium and potassium, and a complete blood count.").toDF("text")
val res = pipeline.fit(data).transform(data)
Results
| ner_chunk | entity | LOINC Code | Resolution | all_k_results | all_k_cosine_distances | all_k_resolutions |
|:----------------------|:---------|:-------------|:----------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| glucose | Test | 2345-7 | glucose [Glucose [Mass/volume] in Serum or Plasma] | 2345-7:::47621-8:::50016-5:::104638-2:::6883-3:::4269-7:::72648-9:::41104-1:::LP... | 0.0000:::0.1485:::0.1675:::0.1755:::0.1831:::0.2043:::0.2341:::0.2474:::0.2676::... | glucose [Glucose [Mass/volume] in Serum or Plasma]:::Glucose IV [Glucose.IV [Mas... |
| hemoglobin A1c levels | Test | 41995-2 | hemoglobin a1c [Hemoglobin A1c [Mass/volume] in Blood] | 41995-2:::4548-4:::51196-4:::4547-6:::112870-1:::96595-4:::17856-6:::74246-0:::6... | 0.0811:::0.1184:::0.1908:::0.2340:::0.2511:::0.2618:::0.2672:::0.2739:::0.2786::... | hemoglobin a1c [Hemoglobin A1c [Mass/volume] in Blood]:::Hgb A1c [Hemoglobin A1c... |
| basic metabolic panel | Test | 51990-0 | basic metabolic panel [Basic metabolic panel - Blood] | 51990-0:::24320-4:::24321-2:::24323-8:::24322-0:::89044-2:::101655-9:::LP146091-... | 0.0000:::0.0953:::0.1003:::0.1104:::0.1117:::0.1420:::0.1659:::0.2327:::0.2473::... | basic metabolic panel [Basic metabolic panel - Blood]:::basic metabolic 1998 pan... |
| sodium | Test | 2951-2 | sodium [Sodium [Moles/volume] in Serum or Plasma] | 2951-2:::LP31577-7:::23915-2:::24326-1:::19096-7:::75264-2:::81011-9:::23611-7::... | 0.0000:::0.1845:::0.2493:::0.3039:::0.3136:::0.3281:::0.3415:::0.3727:::0.3744::... | sodium [Sodium [Moles/volume] in Serum or Plasma]:::saline [Saline method]:::NaC... |
| potassium | Test | 2823-3 | potassium [Potassium [Moles/volume] in Serum or Plasma] | 2823-3:::28003-2:::12812-4:::29349-8:::86919-8:::24326-1:::10322-6:::32552-2:::1... | 0.0000:::0.2364:::0.2837:::0.3011:::0.3013:::0.3081:::0.3169:::0.3174:::0.3188::... | potassium [Potassium [Moles/volume] in Serum or Plasma]:::sodium/potassium [Sodi... |
| complete blood count | Test | 24358-4 | Complete Blood Count [Hemogram without Platelets panel - Blood] | 24358-4:::58410-2:::786-4:::33255-1:::11282-1:::112620-0:::74412-8:::777-3:::490... | 0.0000:::0.1648:::0.2574:::0.2604:::0.2720:::0.2844:::0.2986:::0.3003:::0.3213::... | Complete Blood Count [Hemogram without Platelets panel - Blood]:::complete blood... |
Model Information
| Model Name: | biolordresolve_loinc_2_83 |
| Compatibility: | Healthcare NLP 6.4.1+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [embeddings] |
| Output Labels: | [loinc_code] |
| Language: | en |
| Size: | 878.2 MB |
| Case sensitive: | false |