Spark NLP for Healthcare Release Notes 5.3.3

5.3.3

Highlights

We are delighted to announce remarkable enhancements and updates in our latest release of Spark NLP for Healthcare. This release comes with a brand new PipelineTracer module to return structured jsons from pretrained pipelines, brand new hyperparameters to customize relation extraction models and Deidentification process, and 22 new clinical pretrained models and pipelines.

• Introducing 7 new Sentence Entity Resolver Models for entity mapping to medical terminologies, using SOTA BioLord sentence embeddings
• Clinical document analysis with one-liner pretrained pipelines for specific clinical tasks and concepts
• Introducing 2 new Chunk Mapper models designed for medical code mapping between SNOMED and MedDRA terminologies
• Improved version of Social Determinants of Health (SDoH) named entity recognition model with reduced set of core entities
• Automating pipeline tracing and analysis with PipelineTracer to help return structured jsons from pretrained pipelines via the OuputParser module
• Configuring age-based obfuscation with the setAgeGroups parameter
• Enhancing date obfuscation control with the setKeepYear parameter in the Deidentification annotator to allow year info intact
• Broadening relation extraction with extended scope windows, directionSensitive and filterByTokenDistance parameters to allow further customization and reduce FPs
• Enhancing rule-based annotators with the ner_source field for improved chunk tracking and prioritization
• Introduction of a new parameter dataSetInfo to store dataset details for AssertionDL and GenericClassifier for traceability
• Converting visual NER annotations to CoNLL format for training text-based NER models with visual annotations
• Performance analysis of deidentification pipelines on clinical texts in a cluster environment
• New blogposts on relation extraction, MedDRA response to treatment, and pretrained pipelines.
• Various core improvements; bug fixes, enhanced overall robustness and reliability of Spark NLP for Healthcare
  • Added training params to trainable annotators within the metadata of the trained models
  • Updated Risk Adjustment module with V28Y24
  • Resolved index issue in AssertionChunkConverter annotator and AnnotationLab.get_assertion_data modules
  • Resolved saving issue in Flattener annotator
• Updated notebooks and demonstrations for making Spark NLP for Healthcare easier to navigate and understand
  • New PipelineTracer and PipelineOutputParser Notebook
  • Updated Task Based Clinical Pretrained Pipelines Notebook
  • Updated Pretrained Clinical Pipelines Notebook
  • Updated ADE Demo
  • Updated NER_POSOLOGY Demo
  • Updated NER_RADIOLOGY Demo
  • Updated VOP Demo
  • Updated SDOH Demo
  • Updated ONCOLOGY Demo
• The addition and update of numerous new clinical models and pipelines continue to reinforce our offering in the healthcare domain

These enhancements will elevate your experience with Spark NLP for Healthcare, enabling more efficient, accurate, and streamlined analysis of healthcare-related natural language data.

Introducing 7 New Sentence Entity Resolver Models for Entity Mapping to Medical Terminologies, Using SOTA BioLord Sentence Embeddings

The latest lineup of 7 cutting-edge resolver models are designed to enhance clinical entity mapping and coding accuracy. These models leverage advanced natural language processing to seamlessly map medical entities and concepts to standardized codes, facilitating streamlined data analysis and healthcare decision-making. Each model specializes in precise code assignment within specific medical domains, from drug ingredients to veterinary-related entities. Dive into our resolver models and empower your applications with state-of-the-art clinical entity resolution.

Example:

icd10cm_resolver = SentenceEntityResolverModel.pretrained("biolordresolve_icd10cm_augmented_billable_hcc", "en", "clinical/models") \
    .setInputCols(["sentence_embeddings"])\
    .setOutputCol("icd10_code")\
    .setDistanceFunction("EUCLIDEAN")

text = "John Doe, a 49-year-old male with CMT2P, AIDS-causing virus infection, and PKD2, presents for a follow-up visit to manage his chronic conditions."

Result:

Clinical Document Analysis with One-Liner Pretrained Pipelines for Specific Clinical Tasks and Concepts

We introduce a suite of advanced, hybrid pretrained pipelines, specifically designed to streamline the clinical document analysis process. These pipelines are built upon multiple state-of-the-art (SOTA) pretrained models, delivering a comprehensive solution for quickly extracting vital information.

What sets this release apart is the elimination of complexities typically involved in building and chaining models. Users no longer need to navigate the intricacies of constructing intricate pipelines from scratch or the uncertainty of selecting the most effective model combinations. Our new pretrained pipelines simplify these processes, offering a seamless, user-friendly experience.

Example:

from sparknlp.pretrained import PretrainedPipeline

pipeline_sdoh = PretrainedPipeline("explain_clinical_doc_sdoh", "en", "clinical/models")

text = """The patient reported experiencing symptoms of anxiety and depression, which have been affecting his quality of life. 
He reported a history of childhood trauma related to violence and abuse in his household, which has contributed to his smoking, alcohol use and current mental health struggles."""

NER and Assertion Result:

Relation Extraction Result:

Please check the Task Based Clinical Pretrained Pipelines model for more information

Introducing 2 new Chunk Mapper models designed for medical code mapping between SNOMED and MedDRA terminologies.

Introducing a suite of new ChunkMapper models designed to streamline medical code mapping tasks. These models include mappings between MedDRA and SNOMED codes, offering a comprehensive solution for interoperability within medical systems.

Example:

chunkMapper = ChunkMapperModel.load('meddra_llt_snomed_mapper')\
    .setInputCols(["meddra_llt_code2chunk"])\
    .setOutputCol("mappings")\
    .setRels(["snomed_code"])

text = ["Chronic renal insufficiency", "Gastritis", "Transient ischemic attack"]

Result:

Improved Version of Social Determinants of Health (SDoH) Named Entity Recognition Model with Reduced Set of Core Entities

We are introducing our new Social Determinants of Health (SDoH) Named Entity Recognition model that has been specifically designed to identify and extract entities related to various social determinants of health. This new model is an improvement over our previous version, aimed at better understanding and tracking the impacts of social factors on health.

Example:

ner_model = MedicalNerModel.pretrained("ner_sdoh_core", "en", "clinical/models")\
    .setInputCols(["sentence", "token", "embeddings"])\
    .setOutputCol("ner")

sample_texts = [["""Smith is 55 years old, living in New York, a divorced Mexcian American woman with financial problems. She speaks Spanish and Portuguese. She lives in an apartment. She has been struggling with diabetes for the past 10 years and has recently been experiencing frequent hospitalizations due to uncontrolled blood sugar levels. Smith works as a cleaning assistant and cannot access health insurance or paid sick leave. She has a son, a student at college. Pt with likely long-standing depression. She is aware she needs rehab. Pt reports having her catholic faith as a means of support as well.  She has a long history of etoh abuse, beginning in her teens. She reports she has been a daily drinker for 30 years, most recently drinking beer daily. She smokes a pack of cigarettes a day. She had DUI in April and was due to court this week."""]]

Result:

Please check the model card and SDOH Demo

Automating Pipeline Tracing and Analysis with PipelineTracer to Help Return Structured JSONs from Pretrained Pipelines Via the PipelineOuputParser module

PipelineTracer is a versatile class designed to trace and analyze the stages of a pipeline, offering in-depth insights into entities, assertions, deidentification, classification, and relationships. It also facilitates the creation of parser dictionaries for building a PipelineOutputParser. Key functions include printing the pipeline schema, creating parser dictionaries, and retrieving possible assertions, relations, and entities. Also, provide direct access to parser dictionaries and available pipeline schemas

Please check the PipelineTracer and PipelineOutputParser notebook for more information

PipelineTracer Example:

from sparknlp.pretrained import PretrainedPipeline
from sparknlp_jsl.pipeline_tracer import PipelineTracer

oncology_pipeline = PretrainedPipeline("explain_clinical_doc_oncology", "en", "clinical/models")

pipeline_tracer = PipelineTracer(oncology_pipeline)

column_maps = pipeline_tracer.createParserDictionary()
column_maps.update({"document_identifier": "explain_clinical_doc_oncology"})
print(column_maps)

column_maps Result:

{
    'document_identifier': 'explain_clinical_doc_oncology',
    'document_text': 'document',
    'entities': [
        {
            'ner_chunk_column_name': 'merged_chunk',
            'assertion_column_name': '',
            'resolver_column_name': ''
        },
        {
            'ner_chunk_column_name': 'merged_chunk_for_assertion',
            'assertion_column_name': 'assertion',
            'resolver_column_name': ''
        }
    ],
    'relations': ['all_relations'],
    'summaries': [],
    'deidentifications': [],
    'classifications': []
 }

PipelineOutputParser Example:

text = """The Patient underwent a computed tomography (CT) scan of the abdomen and pelvis, which showed a complex ovarian mass. A Pap smear performed one month later was positive for atypical glandular cells suspicious for adenocarcinoma. The pathologic specimen showed extension of the tumor throughout the fallopian tubes, appendix, omentum, and 5 out of 5 enlarged lymph nodes. The final pathologic diagnosis of the tumor was stage IIIC papillary serous ovarian adenocarcinoma. Two months later, the patient was diagnosed with lung metastases.Neoadjuvant chemotherapy with the regimens of Cyclophosphamide (500 mg/m2) is being given for 6 cycles with poor response"""

results = oncology_pipeline.fullAnnotate()

from sparknlp_jsl.pipeline_output_parser import PipelineOutputParser

pipeline_parser = PipelineOutputParser(column_maps)
result = pipeline_parser.run(results)

PipelineOutputParser Result:

{
    'result': [
        {
            'document_identifier': 'explain_clinical_doc_oncology',
            'document_text': ['The Patient underwent a computed tomography (CT) scan of the abdomen and pelvis, ....'],
            'entities': [
                [{'chunk_id': '0',
                'begin': 24,
                'end': 42,
                'chunk': 'computed tomography',
                'label': 'Imaging_Test',
                'assertion': None,
                'term_code': None},
                {'chunk_id': '1',
                'begin': 45,
                'end': 46,
                'chunk': 'CT',
                'label': 'Imaging_Test',
                'assertion': None,
                'term_code': None},
                ...
                ],
                [{'chunk_id': '0',
                'begin': 24,
                'end': 42,
                'chunk': 'computed tomography',
                'label': 'Imaging_Test',
                'assertion': 'Past',
                'term_code': None},
                {'chunk_id': '1',
                'begin': 45,
                'end': 46,
                'chunk': 'CT',
                'label': 'Imaging_Test',
                'assertion': 'Past',
                'term_code': None}]
            ],
            'relations': [
                [{'relation': 'is_location_of',
                'entity1': 'Site_Other_Body_Part',
                'entity1_begin': '104',
                'entity1_end': '110',
                'chunk1': 'ovarian',
                'entity2': 'Tumor_Finding',
                'entity2_begin': '112',
                'entity2_end': '115',
                'chunk2': 'mass',
                'confidence': '0.922661'},
                {'relation': 'is_finding_of',
                'entity1': 'Pathology_Test',
                'entity1_begin': '120',
                'entity1_end': '128',
                'chunk1': 'Pap smear',
                'entity2': 'Cancer_Dx',
                'entity2_begin': '213',
                'entity2_end': '226',
                'chunk2': 'adenocarcinoma',
                'confidence': '0.52542114'},
                ...]
            ],
            'summaries': [],
            'deidentifications': [],
            'classifications': []
        }
    ]
}

getParserDictDirectly Example:

from sparknlp_jsl.pipeline_tracer import PipelineTracer

columns_directly = PipelineTracer.getParserDictDirectly("clinical_deidentification", "en", "clinical/models")

print(columns_directly)

getParserDictDirectly Result:

{
    'document_identifier': 'clinical_deidentification',
    'document_text': 'sentence',
    'entities': [{
        'ner_chunk_column_name': 'ner_chunk',
        'assertion_column_name': '',
        'resolver_column_name': ''}],
    'relations': [],
    'summaries': [],
    'deidentifications': [{
        'original': 'sentence',
        'obfuscated': 'obfuscated',
        'masked': ''}],
    'classifications': []}

Configuring Age-based Obfuscation with the setAgeGroups Parameter

This method, setAgeGroups, is used in conjunction with the obfuscateByAgeGroups parameter to specify age ranges for obfuscation. If the specified ageGroups dictionary does not cover all ages, the obfuscation defaults to the ageRanges parameter. Each entry in the dictionary includes an age group name paired with a range of two integers: the lower and upper bounds of the age group. By default, the method includes a standard dictionary of age groups in English, but users can customize this dictionary to suit specific age classifications and obfuscation requirements. This method takes a value parameter containing a dictionary mapping age group names to corresponding age ranges for obfuscation.

Example:

obfuscation = DeIdentification()\
    .setInputCols(["sentence", "token", "ner_subentity_chunk"]) \
    .setOutputCol("deidentified") \
    .setMode("obfuscate")\
    .setObfuscateByAgeGroups(True)\
    .setAgeGroups({"baby": [0, 1],
                   "toddler": [1, 4],
                   "child": [4, 13],
                   "teenager": [13, 20],
                   "adult": [20, 65],
                   "senior": [65, 100] })

text ='''
Name: Joseph Brown, Age: 17, Phone: (9) 7765-5632.
This 17 yrs old male, presented with chest heaviness that started during a pick-up basketball game. 
Mark Smith, aged 55, and his daughter (7) Mary were involved in an accident during their travel.
'''

Result:

Enhancing Date Obfuscation Control with the setKeepYear Parameter in Deidentification Annotator to Allow year Info Intact

The setKeepYear parameter to improve date obfuscation controls. This feature allows users to decide whether to retain the year in date entities while obfuscating the month and day. The default setting is False.

• True, the year remains unchanged, ensuring consistency in data that relies on year-specific information.
• False, the entire date, including the year, will be modified.

Example:

obfuscation = DeIdentification()\
    .setInputCols(["sentence", "token", "ner_subentity_chunk"]) \
    .setOutputCol("deidentified") \
    .setMode("obfuscate")\
    .setObfuscateDate(True)\
    .setObfuscateRefSource("faker") \
    .setKeepYear(True)

.setKeepYear(False) Result:

.setKeepYear(True) Result:

As you can see, 2022 has not been changed.

Broadening Relation Extraction with Extended Scope-Windows, directionSensitive and filterByTokenDistance Parameters to Allow Further Customization and Reduce FPs

• In RelationRxtraction, scopeWindow expands beyond the immediate tokens of target chunks. By applying a scopeWindow [X, Y], additional X tokens to the left and Y tokens to the right become crucial for feature generation, enriching contextual information essential for precise embeddings-based feature extraction.

Example:

re_model = RelationExtractionModel.pretrained("re_oncology_wip", "en", "clinical/models") \
    .setInputCols(["embeddings", "pos_tags", "ner_chunk", "dependencies"]) \
    .setOutputCol("re_oncology_results") \
    .setScopeWindow([5,5])

• The features directionSensitive and filterByTokenDistance have been implemented. The directionSensitive setting determines how entity relations are considered. If set to true, only relations in the form of ENTITY1-ENTITY2 are considered. If set to false, both ENTITY1-ENTITY2 and ENTITY2-ENTITY1 relations are considered. The filterByTokenDistance setting is a criterion for filtering based on the number of tokens between entities. The model only identifies relations where the entities are separated by fewer than the specified number of tokens.

directionSensitive Example:

re_ner_chunk_filter = RENerChunksFilter() \
    .setInputCols(["ner_chunk", "dependencies"])\
    .setOutputCol("re_ner_chunk")\
    .setMaxSyntacticDistance(10)\
    .setDirectionSensitive(True)\
    .setRelationPairs(["test-problem", #"problem-test"
                       "treatment-problem", #"problem-treatment"
                       ])\

redl_model = RelationExtractionDLModel.pretrained("redl_clinical_biobert", "en", "clinical/models")\
    .setInputCols(["re_ner_chunk", "sentence"])\
    .setOutputCol("relations")\
    .setPredictionThreshold(0.5)\
    .setRelationPairsCaseSensitive(False)\


text = ''' She was treated with a five-day course of amoxicillin for a respiratory tract infection.
She was on metformin, glipizide, and dapagliflozin for T2DM and additionally atorvastatin and gemfibrozil for HTG.
However, serum chemistry obtained six hours after presentation revealed the anion gap was still elevated at 21, serum bicarbonate was 16 mmol/L, and lipase was 52 U/L.
The β-hydroxybutyrate level was found to be elevated at 5.29 mmol/L - the original sample was centrifuged and the chylomicron layer was removed before analysis due to interference from turbidity caused by lipemia again.
'''

directionSensitive Result:

filterByTokenDistance Example:

re_ner_chunk_filter = RENerChunksFilter() \
    .setInputCols(["ner_chunk", "dependencies"])\
    .setOutputCol("re_ner_chunk")\
    .setMaxSyntacticDistance(10)\
    .setDirectionSensitive(True)\
    .setRelationPairs(["test-problem", #"problem-test"
                       "treatment-problem", #"problem-treatment"
                       ])\
    .setFilterByTokenDistance(4)

redl_model = RelationExtractionDLModel.pretrained("redl_clinical_biobert", "en", "clinical/models")\
    .setInputCols(["re_ner_chunk", "sentence"])\
    .setOutputCol("relations")\
    .setPredictionThreshold(0.5)\
    .setRelationPairsCaseSensitive(False)\

text = ''' She was treated with a five-day course of amoxicillin for a respiratory tract infection.
She was on metformin, glipizide, and dapagliflozin for T2DM and additionally atorvastatin and gemfibrozil for HTG.
However, serum chemistry obtained six hours after presentation revealed the anion gap was still elevated at 21, serum bicarbonate was 16 mmol/L, and lipase was 52 U/L.
The β-hydroxybutyrate level was found to be elevated at 5.29 mmol/L - the original sample was centrifuged and the chylomicron layer was removed before analysis due to interference from turbidity caused by lipemia again.
'''

filterByTokenDistance Result:

please see the blogpost Next-Level Relation Extraction in Healthcare NLP: Introducing New Directional and Contextual Features

Enhancing Rule-Based Annotators with the ner_source Field for Improved Chunk Tracking and Prioritization

Enhancing rule-based annotators such as ContextualParser, TextMatcherInternal, RegexMatcherInternal, and EntityRulerInternal with ner_source field for improved chunk tracking and prioritization

We have enhanced rule-based annotators, including ContextualParser, TextMatcherInternal, RegexMatcherInternal, and EntityRulerInternal, by adding the ner_source field. This improvement allows for better chunk tracking and prioritization, enabling clients to trace the origin of chunks effectively. Additionally, with the ner_source field, NerConverterInternal and ChunkMergerApproach can now prioritize chunks using the .setChunkPrecedence("ner_source") method, leading to more accurate and efficient entity recognition and handling.

Example:

regex_matcher_internal = RegexMatcherInternal()\
    .setInputCols('document')\
    .setStrategy("MATCH_ALL")\
    .setOutputCol("regex_matches")\
    .setExternalRules(path='./rules/regex_rules.txt', delimiter='~')

entityExtractor = TextMatcherInternal()\
    .setInputCols(["document", "token"])\
    .setEntities("matcher_drug.csv")\
    .setOutputCol("matched_text")\
    .setCaseSensitive(False)\
    .setDelimiter("#")\
    .setMergeOverlapping(False)

entityRuler = EntityRulerInternalApproach()\
    .setInputCols(["document", "token"])\
    .setOutputCol("entities")\
    .setPatternsResource("entities.json")\
    .setCaseSensitive(False)\

text = """ Name: John Smith, Record date: 2093-01-13, MR #719435, John's doctor prescribed aspirin for his heart condition, along with paracetamol for his fever and headache, amoxicillin for his tonsilitis."""

Result:

Introduction of a new parameter dataSetInfo to store dataset details for AssertionDL and GenericClassifier for Traceability

The parameters from the Approach class, utilized during model training, have been added into the Model class. These values are now directly stored within the model itself. Additionally, a new parameter named “dataSetInfo”(details regarding the dataset) has been added for AssertionDL and GenericClassifier.

Example:

scope_window = [10,10]

assertionStatus = AssertionDLApproach()\
    .setLabelCol("label")\
    .setInputCols("document", "chunk", "embeddings")\
    .setOutputCol("assertion")\
    .setBatchSize(64)\
    .setDropout(0.1)\
    .setLearningRate(0.001)\
    .setEpochs(5)\
    .setValidationSplit(0.2)\
    .setMaxSentLen(250)\
    ...
    .setDatasetInfo("i2b2_assertion_sample_short_dataset")

# save trained model and load 
clinical_assertion = AssertionDLModel.load("./assertion_custom_model") \
    .setInputCols(["sentence", "ner_chunk", "embeddings"]) \
    .setOutputCol("assertion")

!cat ./assertion_custom_model/metadata/part-00000

Result:

{
    "paramMap": {
        "startCol": "start",
        "inputCols": ["document","chunk","embeddings"],
        "learningRate": 0.0010000000474974513,
        "outputLogsPath": "training_logs/",
        "storageRef": "clinical",
        "maxSentLen": 250,
        "scopeWindow": [10,10],
        "endCol": "end",
        "label": "label",
        "enableOutputLogs": true,
        "batchSize": 64,
        "includeConfidence": true,
        "graphFile": "./tf_graphs/assertion_graph.pb",
        "epochs": 5,
        "dropout": 0.10000000149011612,
        "graphFolder": "./tf_graphs",
        "outputCol": "assertion",
        "validationSplit": 0.20000000298023224,
        "datasetInfo": "i2b2_assertion_sample_short_dataset"
    }
}

Converting Visual NER Annotations to CoNLL Format for Training Text-Based NER Models with Visual Annotations

This module converts Visual NER annotations into the CoNLL format using the JohnSnowLabs NLP Lab. By processing an NLP Lab-exported JSON file containing Visual NER results, it generates a CoNLL file that is suitable for training Named Entity Recognition (NER) models.

Example:

# Import the module
from sparknlp_jsl.alab import AnnotationLab
alab = AnnotationLab()

# Download sample Visual NER result JSON file
!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Annotation_Lab/data/alab_visualner_result.json 

# Convert Visual NER annotations to CoNLL format
df = alab.get_conll_data_from_visualner(
    input_json_path = "alab_visualner_result.json",
    output_name = "visual_ner.conll",
    save_dir  = "exported_conll"
)

Performance Analysis of Deidentification Pipelines on Clinical Texts in a Cluster Environment

• Deidentification Pipelines Benchmarks

  This benchmark provides valuable insights into the efficiency and scalability of deidentification pipelines in different computational environments.

  • Dataset: 100000 Clinical Texts from MTSamples, approx. 508 tokens and 26.44 chunks per text.
  • Versions:[May-2024]
    • spark-nlp Version: v5.3.2
    • spark-nlp-jsl Version: v5.3.2
    • Spark Version: v3.4.0
  • Instance Type:
    • DataBricks Config:
      • 32 CPU Core, 128GiB RAM (8 worker) (2.7 $/hr)\

      data_count	partition	Databricks
      100000	512	1h 42m 55s

    • AWS EC2 instance Config: - 8 CPU cores, 58GiB RAM (r6a.2xlarge $0.4536/h)

      data_count	partition	AWS
      100000	512	3h 3m 40

• Deidentification Pipelines Speed Comparison

  This benchmark presents a detailed comparison of various deidentification pipelines applied to a dataset of 10,000 custom clinical texts, aiming to anonymize sensitive information for research and analysis. The comparison evaluates the elapsed time and processing stages of different deidentification pipelines. Each pipeline is characterized by its unique combination of Named Entity Recognition (NER), deidentification methods, rule-based NER, clinical embeddings, and chunk merging processes.

  • Dataset: 10K Custom Clinical Texts with 1024 partitions, approx. 500 tokens and 14 chunks per text.
  • Versions:
    • spark-nlp Version: v5.3.1
    • spark-nlp-jsl Version: v5.3.1
    • Spark Version: v3.4.0
  • Instance Type:
    • 8 CPU Cores 52GiB RAM (Colab Pro - High RAM)

PS: The reasons pipelines with the same stages have different costs are due to the layers of the NER model and the hardcoded regexes in Deidentification.

Please check Deidentification Benchmarks for more detail

New Blogposts on Relation Extraction, MedDRA, Response to Treatment, and Pretrained Pipelines.

• Next-Level Relation Extraction in Healthcare NLP: Introducing New Directional and Contextual Features
• Clinical Document Analysis with One-Liner Pretrained Pipelines in Healthcare NLP
• Mapping Medical Terms to MedDRA Ontology Using Healthcare NLP
• Response to Cancer Treatment

Various Core Improvements: Bug Fixes, Enhanced Overall Robustness, and Reliability of Spark NLP for Healthcare

• Added training params to trainable annotators within the metadata of the trained models
• Updated Risk Adjustment module with V28Y24
• Resolved index issue in AssertionChunkConverter annotator and AnnotationLab.get_assertion_data modules
• Resolved saving issue in Flattener annotator

Updated Notebooks And Demonstrations For making Spark NLP For Healthcare Easier To Navigate And Understand

• New PipelineTracer and PipelineOutputParser Notebook
• Updated Task Based Clinical Pretrained Pipelines Notebook
• Updated Pretrained Clinical Pipelines Notebook
• Updated ADE Demo
• Updated NER_POSOLOGY Demo
• Updated NER_RADIOLOGY Demo
• Updated VOP Demo
• Updated SDOH Demo
• Updated ONCOLOGY Demo

We Have Added And Updated A Substantial Number Of New Clinical Models And Pipelines, Further Solidifying Our Offering In The Healthcare Domain.

• meddra_llt_snomed_mapper
• snomed_meddra_llt_mapper
• explain_clinical_doc_sdoh
• explain_clinical_doc_oncology
• explain_clinical_doc_granular
• explain_clinical_doc_mental_health
• ner_medication_generic_pipeline
• ner_deid_context_augmented_pipeline
• ner_deid_generic_context_augmented_pipeline
• ner_deid_subentity_context_augmented_pipeline
• biolordresolve_rxnorm_augmented
• biolordresolve_umls_general_concepts
• biolordresolve_icd10cm_augmented_billable_hcc
• sbiobertresolve_snomed_veterinary_wip
• sbiobertresolve_umls_general_concepts
• biolordresolve_avg_rxnorm_augmented
• biolordresolve_snomed_findings_aux_concepts
• biolordresolve_cpt_procedures_measurements_augmented
• sbiobertresolve_umls_disease_syndrome
• sbiobertresolve_umls_findings
• sbiobertresolve_umls_major_concepts
• sbiobertresolve_umls_clinical_drugs
• sbiobertresolve_umls_drug_substance
• sbiobertresolve_icd9

For all Spark NLP for Healthcare models, please check: Models Hub Page

Versions