Detect Assertion Status (assertion_ml_en)


Logistic regression based named entity recognition model for assertions.

Predicted Labels

Hypothetical, Present, Absent, Possible, Conditional, Associated_with_someone_else

Open in Colab Download

How to use

Use as part of an nlp pipeline with the following stages: DocumentAssembler, SentenceDetector, Tokenizer, WordEmbeddingsModel, NerDLModel, NerConverter, AssertionLogRegModel.

clinical_assertion = AssertionDLModel.pretrained("assertion_ml", "en", "clinical/models") \
    .setInputCols(["sentence", "ner_chunk", "embeddings"]) \
nlpPipeline = Pipeline(stages=[documentAssembler, sentenceDetector, tokenizer, word_embeddings, clinical_ner, ner_converter, clinical_assertion])

model =[[""]]).toDF("text"))

light_result = LightPipeline(model).fullAnnotate('Patient has a headache for the last 2 weeks and appears anxious when she walks fast. No alopecia noted. She denies pain')[0]

val clinical_assertion_ml = AssertionLogRegModel.pretrained("assertion_ml", "en", "clinical/models") \
    .setInputCols(["sentence", "ner_chunk", "embeddings"]) \

val pipeline = new Pipeline().setStages(Array(clinical_assertion_ml))

val result =[String].toDS.toDF("text")).transform(data)


The output is a dataframe with a sentence per row and an “assertion” column containing all of the assertion labels in the sentence. The assertion column also contains assertion character indices, and other metadata. To get only the entity chunks and assertion labels, without the metadata, select “ner_chunk.result” and “assertion.result” from your output dataframe.

|   | chunks     | entities | assertion   |
| 0 | a headache | PROBLEM  | present     |
| 1 | anxious    | PROBLEM  | conditional |
| 2 | alopecia   | PROBLEM  | absent      |
| 3 | pain       | PROBLEM  | absent      |

Model Information

Model Name: assertion_ml_en_2.4.0_2.4
Type: ner
Compatibility: Spark NLP 2.4.0+
Edition: Official
License: Licensed
Input Labels: [sentence, ner_chunk, embeddings]
Output Labels: [assertion]
Language: [en]
Case sensitive: false

Data Source

Trained on 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text with ‘embeddings_clinical’.