JSL_MedM (LLM - q4)

Description

This LLM model is trained to perform Q&A, Summarization, RAG, and Chat.

Copy S3 URI

How to use


document_assembler = DocumentAssembler()\
    .setInputCol("text")\
    .setOutputCol("document")

medical_llm = MedicalLLM.pretrained("jsl_medm_q4_v1", "en", "clinical/models")\
    .setInputCols("document")\
    .setOutputCol("completions")\
    .setBatchSize(1)\
    .setNPredict(100)\
    .setUseChatTemplate(True)\
    .setTemperature(0)


pipeline = Pipeline(
    stages = [
        document_assembler,
        medical_llm
])

medm_prompt = """
summarize the following content.

 content:
 ---------------------------- INDICATIONS AND USAGE ---------------------------
 KISUNLA is an amyloid beta-directed antibody indicated for the
 treatment of Alzheimer’s disease. Treatment with KISUNLA should be
 initiated in patients with mild cognitive impairment or mild dementia
 stage of disease, the population in which treatment was initiated in the
 clinical trials. (1)
 ------------------------DOSAGE AND ADMINISTRATION-----------------------
 • Confirm the presence of amyloid beta pathology prior to initiating
 treatment. (2.1)
 • The recommended dosage of KISUNLA is 700 mg administered as
 an intravenous infusion over approximately 30 minutes every four
 weeks for the first three doses, followed by 1400 mg every four
 weeks. (2.2)
 • Consider stopping dosing with KISUNLA based on reduction of
 amyloid plaques to minimal levels on amyloid PET imaging. (2.2)
 • Obtain a recent baseline brain MRI prior to initiating treatment.
 (2.3, 5.1)
 • Obtain an MRI prior to the 2nd, 3rd, 4th, and 7th infusions. If
 radiographically observed ARIA occurs, treatment
 recommendations are based on type, severity, and presence of
 symptoms. (2.3, 5.1)
 • Dilution to a final concentration of 4 mg/mL to 10 mg/mL with 0.9%
 Sodium Chloride Injection, is required prior to administration. (2.4)
 ----------------------DOSAGE FORMS AND STRENGTHS---------------------
 Injection: 350 mg/20 mL (17.5 mg/mL) in a single-dose vial. (3)
 ------------------------------- CONTRAINDICATIONS ------------------------------
 KISUNLA is contraindicated in patients with known serious
 hypersensitivity to donanemab-azbt or to any of the excipients. (4, 5.2)
 ------------------------WARNINGS AND PRECAUTIONS-----------------------
 • Amyloid Related Imaging Abnormalities (ARIA): Enhanced clinical
 vigilance for ARIA is recommended during the first 24 weeks of
 treatment with KISUNLA. Risk of ARIA, including symptomatic
 ARIA, was increased in apolipoprotein E ε4 (ApoE ε4)
 homozygotes compared to heterozygotes and noncarriers. The risk
 of ARIA-E and ARIA-H is increased in KISUNLA-treated patients
 with pretreatment microhemorrhages and/or superficial siderosis. If
 a patient experiences symptoms suggestive of ARIA, clinical
 evaluation should be performed, including MRI scanning if
 indicated. (2.3, 5.1)
 • Infusion-Related Reactions: The infusion rate may be reduced, or
 the infusion may be discontinued, and appropriate therapy initiated
 as clinically indicated. Consider pre-treatment with antihistamines,
 acetaminophen, or corticosteroids prior to subsequent dosing. (5.3)
 -------------------------------ADVERSE REACTIONS------------------------------
 Most common adverse reactions (at least 10% and higher incidence
 compared to placebo): ARIA-E, ARIA-H microhemorrhage, ARIA-H
 superficial siderosis, and headache. (6.1)
"""

data = spark.createDataFrame([[medm_prompt]]).toDF("text")

results = pipeline.fit(data).transform(data)

results.select("completions").show(truncate=False)


val document_assembler = new DocumentAssembler()
    .setInputCol("text")
    .setOutputCol("document")

val medical_llm = MedicalLLM.pretrained("jsl_medm_q4_v1", "en", "clinical/models")
    .setInputCols("document")
    .setOutputCol("completions")
    .setBatchSize(1)
    .setNPredict(100)
    .setUseChatTemplate(True)
    .setTemperature(0)


val pipeline = new Pipeline().setStages(Array(
    document_assembler,
    medical_llm
))

val  medm_prompt = """
summarize the following content.

 content:
 ---------------------------- INDICATIONS AND USAGE ---------------------------
 KISUNLA is an amyloid beta-directed antibody indicated for the
 treatment of Alzheimer’s disease. Treatment with KISUNLA should be
 initiated in patients with mild cognitive impairment or mild dementia
 stage of disease, the population in which treatment was initiated in the
 clinical trials. (1)
 ------------------------DOSAGE AND ADMINISTRATION-----------------------
 • Confirm the presence of amyloid beta pathology prior to initiating
 treatment. (2.1)
 • The recommended dosage of KISUNLA is 700 mg administered as
 an intravenous infusion over approximately 30 minutes every four
 weeks for the first three doses, followed by 1400 mg every four
 weeks. (2.2)
 • Consider stopping dosing with KISUNLA based on reduction of
 amyloid plaques to minimal levels on amyloid PET imaging. (2.2)
 • Obtain a recent baseline brain MRI prior to initiating treatment.
 (2.3, 5.1)
 • Obtain an MRI prior to the 2nd, 3rd, 4th, and 7th infusions. If
 radiographically observed ARIA occurs, treatment
 recommendations are based on type, severity, and presence of
 symptoms. (2.3, 5.1)
 • Dilution to a final concentration of 4 mg/mL to 10 mg/mL with 0.9%
 Sodium Chloride Injection, is required prior to administration. (2.4)
 ----------------------DOSAGE FORMS AND STRENGTHS---------------------
 Injection: 350 mg/20 mL (17.5 mg/mL) in a single-dose vial. (3)
 ------------------------------- CONTRAINDICATIONS ------------------------------
 KISUNLA is contraindicated in patients with known serious
 hypersensitivity to donanemab-azbt or to any of the excipients. (4, 5.2)
 ------------------------WARNINGS AND PRECAUTIONS-----------------------
 • Amyloid Related Imaging Abnormalities (ARIA): Enhanced clinical
 vigilance for ARIA is recommended during the first 24 weeks of
 treatment with KISUNLA. Risk of ARIA, including symptomatic
 ARIA, was increased in apolipoprotein E ε4 (ApoE ε4)
 homozygotes compared to heterozygotes and noncarriers. The risk
 of ARIA-E and ARIA-H is increased in KISUNLA-treated patients
 with pretreatment microhemorrhages and/or superficial siderosis. If
 a patient experiences symptoms suggestive of ARIA, clinical
 evaluation should be performed, including MRI scanning if
 indicated. (2.3, 5.1)
 • Infusion-Related Reactions: The infusion rate may be reduced, or
 the infusion may be discontinued, and appropriate therapy initiated
 as clinically indicated. Consider pre-treatment with antihistamines,
 acetaminophen, or corticosteroids prior to subsequent dosing. (5.3)
 -------------------------------ADVERSE REACTIONS------------------------------
 Most common adverse reactions (at least 10% and higher incidence
 compared to placebo): ARIA-E, ARIA-H microhemorrhage, ARIA-H
 superficial siderosis, and headache. (6.1)
"""

val data = Seq(medm_prompt).toDF("text")

val results = pipeline.fit(data).transform(data)

results.select("completions").show(truncate=False)

Results


KISUNLA is an amyloid beta-directed antibody indicated for the treatment of Alzheimer's disease. It is recommended to initiate treatment in patients with mild cognitive impairment or mild dementia stage of disease. The recommended dosage is 700 mg administered as an intravenous infusion over approximately 30 minutes every four weeks for the first three doses, followed by 1400 mg every four weeks. Patients should have a recent baseline brain MRI prior to initiating treatment and obtain an MRI prior to the 2nd,

Model Information

Model Name: jsl_medm_q4_v1
Compatibility: Healthcare NLP 5.5.0+
License: Licensed
Edition: Official
Language: en
Size: 4.8 GB

Benchmarking

We have generated a total of 400 questions, 100 from each category. These questions were labeled and reviewed by 3 physician annotators. % indicates the preference rate. Please see the more benchmark information here.

## Overall
| Model    | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.29         | 0.25                 | 0.50          |
| ChatGPT  | 0.21         | 0.30                 | 0.26          |
| Neutral  | 0.43         | 0.38                 | 0.17          |
| None     | 0.07         | 0.07                 | 0.08          |
| total    | 1.00         | 1.00                 | 1.00          |

## Summary 
| Model    | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.42         | 0.42                 | 0.50          |
| GPT4o    | 0.33         | 0.33                 | 0.28          |
| Neutral  | 0.17         | 0.17                 | 0.12          |
| None     | 0.08         | 0.08                 | 0.10          |
| Total    | 1.00         | 1.00                 | 1.00          |

## QA

| Model    | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.40         | 0.36                 | 0.60          |
| GPT4o    | 0.15         | 0.19                 | 0.19          |
| Neutral  | 0.38         | 0.38                 | 0.11          |
| None     | 0.08         | 0.08                 | 0.09          |
| Total    | 1.00         | 1.00                 | 1.00          |


## BioMedical

| Model    | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.22         | 0.14                 | 0.55          |
| GPT4o    | 0.21         | 0.36                 | 0.23          |
| Neutral  | 0.49         | 0.44                 | 0.14          |
| None     | 0.07         | 0.06                 | 0.07          |
| Total    | 1.00         | 1.00                 | 1.00          |

## OpenEnded

| Model    | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.21         | 0.19                 | 0.38          |
| GPT4o    | 0.18         | 0.30                 | 0.31          |
| Neutral  | 0.55         | 0.46                 | 0.26          |
| None     | 0.05         | 0.05                 | 0.06          |
| Total    | 1.00         | 1.00                 | 1.00          |