Description
This model maps clinical entities and concepts (like drugs/ingredients) to RxNorm codes using sbiobert_base_cased_mli_onnx Sentence Bert Embeddings. It also returns drug concept classes in the all_k_aux_labels column.
Live Demo Open in Colab Copy S3 URI
How to use
documentAssembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")\
.setInputCols(["sentence","token"])\
.setOutputCol("word_embeddings")
ner_model = MedicalNerModel.pretrained("ner_posology_greedy","en","clinical/models")\
.setInputCols(["sentence","token","word_embeddings"])\
.setOutputCol("ner")
ner_converter = NerConverterInternal()\
.setInputCols(["sentence","token","ner"])\
.setOutputCol("ner_chunk")\
.setWhiteList(["DRUG"])
chunk2doc = Chunk2Doc()\
.setInputCols("ner_chunk")\
.setOutputCol("ner_chunk_doc")
embedder = BertSentenceEmbeddings.pretrained("sbiobert_base_cased_mli_onnx","en","clinical/models")\
.setInputCols(["ner_chunk_doc"])\
.setOutputCol("sentence_embeddings")\
.setCaseSensitive(False)
resolver = SentenceEntityResolverModel.pretrained("sbiobertresolve_rxnorm_augmented_v2","en","clinical/models")\
.setInputCols(["sentence_embeddings"])\
.setOutputCol("rxnorm_code")\
.setDistanceFunction("EUCLIDEAN")
pipeline = Pipeline(stages=[
documentAssembler, sentenceDetector, tokenizer, word_embeddings,
ner_model, ner_converter, chunk2doc, embedder, resolver
])
data = spark.createDataFrame([["The patient was started on metformin 500 mg twice daily for type 2 diabetes and continued on Lipitor for hyperlipidemia. She takes Tylenol PRN for headaches and was prescribed amoxicillin 500 mg for a sinus infection."]]).toDF("text")
result = pipeline.fit(data).transform(data)
documentAssembler = nlp.DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentenceDetector = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = nlp.Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
word_embeddings = nlp.WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")\
.setInputCols(["sentence","token"])\
.setOutputCol("word_embeddings")
ner_model = medical.NerModel.pretrained("ner_posology_greedy","en","clinical/models")\
.setInputCols(["sentence","token","word_embeddings"])\
.setOutputCol("ner")
ner_converter = medical.NerConverterInternal()\
.setInputCols(["sentence","token","ner"])\
.setOutputCol("ner_chunk")\
.setWhiteList(["DRUG"])
chunk2doc = nlp.Chunk2Doc()\
.setInputCols("ner_chunk")\
.setOutputCol("ner_chunk_doc")
embedder = nlp.BertSentenceEmbeddings.pretrained("sbiobert_base_cased_mli_onnx","en","clinical/models")\
.setInputCols(["ner_chunk_doc"])\
.setOutputCol("sentence_embeddings")\
.setCaseSensitive(False)
resolver = medical.SentenceEntityResolverModel.pretrained("sbiobertresolve_rxnorm_augmented_v2","en","clinical/models")\
.setInputCols(["sentence_embeddings"])\
.setOutputCol("rxnorm_code")\
.setDistanceFunction("EUCLIDEAN")
pipeline = nlp.Pipeline(stages=[
documentAssembler, sentenceDetector, tokenizer, word_embeddings,
ner_model, ner_converter, chunk2doc, embedder, resolver
])
data = spark.createDataFrame([["The patient was started on metformin 500 mg twice daily for type 2 diabetes and continued on Lipitor for hyperlipidemia. She takes Tylenol PRN for headaches and was prescribed amoxicillin 500 mg for a sinus infection."]]).toDF("text")
result = pipeline.fit(data).transform(data)
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val sentenceDetector = SentenceDetectorDLModel
.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")
.setInputCols(Array("document"))
.setOutputCol("sentence")
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")
val word_embeddings = WordEmbeddingsModel
.pretrained("embeddings_clinical", "en", "clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("word_embeddings")
val ner_model = MedicalNerModel
.pretrained("ner_posology_greedy", "en", "clinical/models")
.setInputCols(Array("sentence", "token", "word_embeddings"))
.setOutputCol("ner")
val ner_converter = new NerConverterInternal()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")
.setWhiteList(Array("DRUG"))
val chunk2doc = new Chunk2Doc()
.setInputCols("ner_chunk")
.setOutputCol("ner_chunk_doc")
val embedder = BertSentenceEmbeddings
.pretrained("sbiobert_base_cased_mli_onnx", "en","clinical/models")
.setInputCols(Array("ner_chunk_doc"))
.setOutputCol("sentence_embeddings")
.setCaseSensitive(false)
val resolver = SentenceEntityResolverModel
.pretrained("sbiobertresolve_rxnorm_augmented_v2", "en", "clinical/models")
.setInputCols(Array("sentence_embeddings"))
.setOutputCol("rxnorm_code")
.setDistanceFunction("EUCLIDEAN")
val pipeline = new Pipeline().setStages(Array(
documentAssembler, sentenceDetector, tokenizer, word_embeddings,
ner_model, ner_converter, chunk2doc, embedder, resolver
))
val data = Seq("The patient was started on metformin 500 mg twice daily for type 2 diabetes and continued on Lipitor for hyperlipidemia. She takes Tylenol PRN for headaches and was prescribed amoxicillin 500 mg for a sinus infection.").toDF("text")
val res = pipeline.fit(data).transform(data)
Results
| ner_chunk | entity | rxnorm_code | resolution | all_k_results | all_k_distances | all_k_cosine_distances | all_k_resolutions | all_k_aux_labels |
|:-------------------|:---------|--------------:|:----------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| metformin 500 mg | DRUG | 316256 | metformin 500 mg [metformin 500 mg] | 316256:::860974:::861007:::861001:::876009:::861017:::875864:::1807917:::1807915... | 0.0000:::5.3125:::5.5184:::5.8876:::5.9882:::6.1411:::6.1761:::6.3602:::6.4482::... | 0.0000:::0.0462:::0.0496:::0.0569:::0.0589:::0.0619:::0.0629:::0.0662:::0.0679::... | metformin 500 mg [metformin 500 mg]:::metformin hydrochloride 500 mg [metformin ... | Clinical Drug Comp:::Clinical Drug Comp:::Clinical Drug:::Branded Drug Comp:::Br... |
| Lipitor | DRUG | 153165 | lipitor [lipitor] | 153165:::615906:::1111103:::6406:::202999:::218027:::2682565:::1372541:::1037264... | 0.0000:::6.4316:::7.1483:::7.1669:::7.2700:::7.4516:::7.6211:::7.8783:::7.9833::... | 0.0000:::0.0689:::0.0868:::0.0864:::0.0887:::0.0923:::0.0971:::0.1047:::0.1024::... | lipitor [lipitor]:::lipofen [lipofen]:::lipiarmicin [fidaxomicin]:::lipase [lipa... | Brand Name:::Brand Name:::Ingredient:::Ingredient:::Brand Name:::Brand Name:::Br... |
| Tylenol PRN | DRUG | 202433 | tylenol [tylenol] | 202433:::2380832:::1368180:::220562:::2700702:::541837:::89552:::236508:::221152... | 6.9398:::7.5941:::7.7717:::7.8327:::7.9769:::8.0105:::8.0540:::8.1701:::8.1907::... | 0.0862:::0.1063:::0.1141:::0.1140:::0.1164:::0.1152:::0.1213:::0.1274:::0.1253::... | tylenol [tylenol]:::terpinolene [terpinolene]:::terpineol [terpineol]:::tycolene... | Brand Name:::Ingredient:::Ingredient:::Brand Name:::Brand Name:::Brand Name:::In... |
| amoxicillin 500 mg | DRUG | 317616 | amoxicillin 500 mg [amoxicillin 500 mg] | 317616:::565640:::565639:::565643:::563921:::565641:::565638:::575262:::565314::... | 0.0000:::3.3695:::3.4219:::3.4943:::3.5369:::3.6493:::3.6677:::3.8616:::3.9649::... | 0.0000:::0.0193:::0.0200:::0.0207:::0.0213:::0.0227:::0.0229:::0.0255:::0.0271::... | amoxicillin 500 mg [amoxicillin 500 mg]:::amoxicillin 500 mg [amrit] [amoxicilli... | Clinical Drug Comp:::Branded Drug Comp:::Branded Drug Comp:::Branded Drug Comp::... |
Model Information
| Model Name: | sbiobertresolve_rxnorm_augmented_v2 |
| Compatibility: | Healthcare NLP 6.4.0+ |
| License: | Licensed |
| Edition: | Official |
| Input Labels: | [sentence_embeddings] |
| Output Labels: | [rxnorm_code] |
| Language: | en |
| Size: | 1.4 GB |
| Case sensitive: | false |