ICD10CM Entity Resolver

Description

Entity Resolution model Based on KNN using Word Embeddings + Word Movers Distance

Predicted Entities

ICD10-CM Codes and their normalized definition with clinical_embeddings.

Live Demo Open in Colab Download

How to use

...
icd10cm_resolution = ChunkEntityResolverModel.pretrained("chunkresolve_icd10cm_clinical", "en", "clinical/models") \
  .setInputCols(["ner_token", "chunk_embeddings"]) \
  .setOutputCol("icd10cm_code") \
  .setDistanceFunction("COSINE")  \
  .setNeighbours(5)

pipeline_icd10cm = Pipeline(stages = [documentAssembler, sentenceDetector, tokenizer, word_embeddings, clinical_ner, ner_converter, chunk_embeddings, chunk_tokenizer, icd10cm_resolution])

pipeline_model = pipeline_icd10cm.fit(spark.createDataFrame([["""A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and subsequent type two diabetes mellitus (T2DM), one prior episode of HTG-induced pancreatitis three years prior to presentation, associated with an acute hepatitis, and obesity with a body mass index (BMI) of 33.5 kg/m2, presented with a one-week history of polyuria, polydipsia, poor appetite, and vomiting. Two weeks prior to presentation, she was treated with a five-day course of amoxicillin for a respiratory tract infection. She was on metformin, glipizide, and dapagliflozin for T2DM and atorvastatin and gemfibrozil for HTG."""]]).toDF("text"))

result = pipeline_model.transform(data)
...
val icd10cm_resolution = ChunkEntityResolverModel.pretrained("chunkresolve_icd10cm_clinical", "en", "clinical/models") 
  .setInputCols("ner_token", "chunk_embeddings") 
  .setOutputCol("icd10cm_code") 
  .setDistanceFunction("COSINE")  
  .setNeighbours(5)
  
val pipeline = new Pipeline().setStages(Array(documentAssembler, sentenceDetector, tokenizer, word_embeddings, clinical_ner, ner_converter, chunk_embeddings, chunk_tokenizer, icd10cm_resolution))

val result = pipeline.fit(Seq.empty["""A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and subsequent type two diabetes mellitus (T2DM), one prior episode of HTG-induced pancreatitis three years prior to presentation, associated with an acute hepatitis, and obesity with a body mass index (BMI) of 33.5 kg/m2, presented with a one-week history of polyuria, polydipsia, poor appetite, and vomiting. Two weeks prior to presentation, she was treated with a five-day course of amoxicillin for a respiratory tract infection. She was on metformin, glipizide, and dapagliflozin for T2DM and atorvastatin and gemfibrozil for HTG."""].toDS.toDF("text")).transform(data)

Results

|   | chunk                       | entity    | resolved_text                                      | code   | cms                                               |
|---|-----------------------------|-----------|----------------------------------------------------|--------|---------------------------------------------------|
| 0 | T2DM),                      | PROBLEM   | Type 2 diabetes mellitus with diabetic nephrop...  | E1121  | Type 2 diabetes mellitus with diabetic nephrop... |
| 1 | T2DM                        | PROBLEM   | Type 2 diabetes mellitus with diabetic nephrop...  | E1121  | Type 2 diabetes mellitus with diabetic nephrop... |
| 2 | polydipsia                  | PROBLEM   | Polydipsia                                         | R631   | Polydipsia:::Anhedonia:::Galactorrhea             |
| 3 | interference from turbidity | PROBLEM   | Non-working side interference                      | M2656  | Non-working side interference:::Hemoglobinuria... |
| 4 | polyuria                    | PROBLEM   | Other polyuria                                     | R358   | Other polyuria:::Polydipsia:::Generalized edem... |
| 5 | lipemia                     | PROBLEM   | Glycosuria                                         | R81    | Glycosuria:::Pure hyperglyceridemia:::Hyperchy... |
| 6 | starvation ketosis          | PROBLEM   | Propionic acidemia                                 | E71121 | Propionic acidemia:::Bartter's syndrome:::Hypo... |
| 7 | HTG                         | PROBLEM   | Pure hyperglyceridemia                             | E781   | Pure hyperglyceridemia:::Familial hypercholest... |

Model Information

Model Name: chunkresolve_icd10cm_clinical
Compatibility: Spark NLP for Healthcare 3.0.0+
License: Licensed
Edition: Official
Input Labels: [token, chunk_embeddings]
Output Labels: [icd10cm]
Language: en