Integrating a Large Language Model Into Sepsis Predictive Tool Improves Clinical Acceptance Without Delaying Antibiotics in the Emergency Department

The use of artificial intelligence (AI), and more recently, large language models (LLM) afford emergency providers novel tools to better identify patients at risk of sepsis. We previously implemented COMPOSER, an AI model to predict sepsis 4-6 hours prior to clinical manifestation. Here, we test the hypothesis that the addition of an LLM to ingest clinical notes and refine predictions in cases of diagnostic uncertainty to COMPOSER would improve clinical acceptance without impacting timeliness of antibiotics.

We employed a before-after quasi-experimental design to compare the clinical impact of transitioning from an existing sepsis prediction model (COMPOSER) to an LLM-augmented model (COMPOSER-LLM) at two emergency departments (ED) within an academic health system. COMPOSER was used during the pre-intervention period (12/2022-7/2024) and COMPOSER-LLM was used during the post-intervention period (8/2024-3/2025). The outcome of interest was the proportion of alerts in which the clinician selected “no Infection suspected” in each study period as a proxy for model utility and acceptance. We used a Bayesian structural time-series model to a conduct causal impact analysis, adjusting for seasonal effects and confounders (comorbidities and ED volume). The primary outcome was the difference between observed and estimated responses in the intervention period. Additionally, we measured time to antibiotics in both periods to assess if delays in antibiotic administration occurred after the implementation of COMPOSER-LLM.

There were 2539 alert events in the pre-intervention period and 1978 in the post-intervention period. We found a significant drop in rates of “No Infection Suspected” after implementation when comparing the actual vs expected responses in the post-intervention period (25%-30% pre-intervention to 18-22% and post-intervention, with deviations of −5% to −15%, indicating lower observed rates than predicted in the post-intervention period (p < 0.001). Timeliness of antibiotics was not changed after implementation of COMPOSER-LLM (median 140.4 min [IQR 37.3–249.1] to 112.1 min [IQR 32.1–280.3], not significant).

Augmenting a sepsis prediction model with an LLM resulted in a significant decrease in cases where providers selected “no infection suspected,” without negatively impacting antibiotic timeliness.