Description
This LLM model is trained to perform Q&A, Summarization, RAG, and Chat.
Predicted Entities
How to use
from sparknlp_jsl.llm import LLMLoader
llm_loader_pretrained = LLMLoader(spark).pretrained("jsl_medm_q8_v1", "en", "clinical/models")
prompt = """
A 23-year-old pregnant woman at 22 weeks gestation presents with burning upon urination. She states it started 1 day ago and has been worsening despite drinking more water and taking cranberry extract. She otherwise feels well and is followed by a doctor for her pregnancy. Her temperature is 97.7°F (36.5°C), blood pressure is 122/77 mmHg, pulse is 80/min, respirations are 19/min, and oxygen saturation is 98% on room air. Physical exam is notable for an absence of costovertebral angle tenderness and a gravid uterus.
Which of the following is the best treatment for this patient?
A: Ampicillin
B: Ceftriaxone
C: Ciprofloxacin
D: Doxycycline
E: Nitrofurantoin
"""
response = llm_loader_pretrained.generate(prompt)
import com.johnsnowlabs.ml.gguf.LLMLoader
import com.johnsnowlabs.nlp.SparkAccessor.spark
val llmLoader = new LLMLoader().setSparkSession(spark).pretrained("jsl_medm_q8_v1", "en", "clinical/models")
val prompt = """A 23-year-old pregnant woman at 22 weeks gestation presents with burning upon urination. She states it started 1 day ago and has been worsening despite drinking more water and taking cranberry extract. She otherwise feels well and is followed by a doctor for her pregnancy. Her temperature is 97.7°F (36.5°C), blood pressure is 122/77 mmHg, pulse is 80/min, respirations are 19/min, and oxygen saturation is 98% on room air. Physical exam is notable for an absence of costovertebral angle tenderness and a gravid uterus.
Which of the following is the best treatment for this patient?
A: Ampicillin
B: Ceftriaxone
C: Ciprofloxacin
D: Doxycycline
E: Nitrofurantoin""".stripMargin
val response = llmLoader.generate(prompt)
Results
"""The correct answer is E: Nitrofurantoin.
The patient is presenting with symptoms of urinary tract infection (UTI), which is common during pregnancy. Nitrofurantoin is a first-line antibiotic for uncomplicated UTI during pregnancy. It is safe and effective in treating UTI during pregnancy and has been used for many years without any adverse effects on the fetus.
"""
Model Information
| Model Name: | jsl_medm_q8_v1 |
| Compatibility: | Healthcare NLP 5.4.0+ |
| License: | Licensed |
| Edition: | Official |
| Language: | en |
| Size: | 7.7 GB |
Benchmarking
We have generated a total of 400 questions, 100 from each category. These questions were labeled and reviewed by 3 physician annotators. % indicates the preference rate
## Overall
| Model | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.29 | 0.25 | 0.50 |
| ChatGPT | 0.21 | 0.30 | 0.26 |
| Neutral | 0.43 | 0.38 | 0.17 |
| None | 0.07 | 0.07 | 0.08 |
| total | 1.00 | 1.00 | 1.00 |
## Summary
| Model | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.42 | 0.42 | 0.50 |
| GPT4o | 0.33 | 0.33 | 0.28 |
| Neutral | 0.17 | 0.17 | 0.12 |
| None | 0.08 | 0.08 | 0.10 |
| Total | 1.00 | 1.00 | 1.00 |
## QA
| Model | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.40 | 0.36 | 0.60 |
| GPT4o | 0.15 | 0.19 | 0.19 |
| Neutral | 0.38 | 0.38 | 0.11 |
| None | 0.08 | 0.08 | 0.09 |
| Total | 1.00 | 1.00 | 1.00 |
## BioMedical
| Model | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.22 | 0.14 | 0.55 |
| GPT4o | 0.21 | 0.36 | 0.23 |
| Neutral | 0.49 | 0.44 | 0.14 |
| None | 0.07 | 0.06 | 0.07 |
| Total | 1.00 | 1.00 | 1.00 |
## OpenEnded
| Model | Factuality % | Clinical Relevancy % | Conciseness % |
|----------|--------------|----------------------|---------------|
| JSL-MedM | 0.21 | 0.19 | 0.38 |
| GPT4o | 0.18 | 0.30 | 0.31 |
| Neutral | 0.55 | 0.46 | 0.26 |
| None | 0.05 | 0.05 | 0.06 |
| Total | 1.00 | 1.00 | 1.00 |
PREVIOUSJSL_MedM (LLM - q4)