But, this led a group to investigate: just how uncaring are various LLMs? Do different LLMs have different levels of “safety” vs. “resource” balance? And the answer is – yes:

They tossed a thousand clinical vignettes into the various models and determined their “under-triage”(FN) to "over-triage”(FP) ratios out-of-the-box. It turns out, interestingly, GPT-5-Mini (which these authors also found was most aligned with ChatGPT Health) was actually one of the “most safe”.

But, in further fascinating experiments, these authors explicitly prompted the LLMs to take into account cost functions for missed under-triage and apply decision analysis principles to their output. No math changed hands, just a simple prompt to “do it”. It turns out, by prompting various cost-functions along with the clinical case, this framing actually changes the triaging behavior of the LLM!

That’s sort of the “good news”. The bad news – well, since it’s just a prompt dropping lottery balls down into the black hole of “reasoning”, the models themselves don’t necessarily steer as directed, and the extent to which directed cost functions affect outputs does not reliably correlate:

So, no, this sort of “cost function coaching” is probably not a reliable and generally replicable solution to the problem of clinical safety. It is far better to design or tune a model and its self-evaluation harness explicitly, but these are still fascinating insights.

Rapid response teams – and even disease-specific response teams (pulmonary embolism response teams!) – have become generally ubiquitous over the past decades. The evidence supporting their benefit is not nearly as strong as proponents would suggest, consisting primarily of observational and uncontrolled before-and-after observations – and harms and costs are rarely mentioned.

So, let’s pair an RRT notification alert with an equally-suspicious early-warning score, the Epic Deterioration Index –

– and watch the mortality go down:

I have two major questions to address here:

First, the magnitude of the effect size is implausible. A 4.5% absolute (~14% adjusted relative) reduction in mortality is profound. This article is describing an intervention resulting in hundreds more of the highest-risk patients making it out of their hospitals. A mere 12% absolute increase in assessments means the excess RRT activations saved, effectively, 1 in 3 of the additional patients they were called upon – or else there are other factors at play unrelated to the intervention. The medical literature is littered with similarly optimistic findings that haven’t nearly panned out, particularly this type of before-and-after observational intervention.

Furthermore, there are massive holes in the causal chain. An alert itself is not a life-saving intervention. An RRT activation is not a life-saving intervention. Mortality reductions result from specific changes in patient management. This article does not describe how patients’ trajectories were altered to avoid these (apparently) preventable deaths. The implication here is these assessments must have clarified a missed diagnosis, profoundly altered treatment, and elicited a dramatic response with the new approach.

In a world where early warning systems and RRT systems each have weak supporting evidence, there is not much reason to give these findings substantial weight at all.

But, whereas the last look at nirsevimab covered the first season of RSV exposure after birth, this publication attempts to do an analysis of the “second season” – and it’s always fun to have a directed acyclic graph!

The key behind showing this being the complexity and challenge of attributing causal effects to an intervention in these sorts of retrospective case-control studies.

So, why even look at this if there are all these limitations? Because, unfortunately, whilst academic purists have the luxury of downplaying these data, these confounded and limited observations are frequently all health systems have to work with when making funding decisions. And these data certainly make an interesting case for extended durability of the protection from nirsevimab:

It’s probably safe to say nirsevimab is preferred over palivizumab, and the better question is how widely prophylactic vaccination programs ought be instituted. This sort of persistent protection adds to its appeal.

And, perhaps, we’re seeing a shift – where compensation is provided in some fashion, including cold, hard, GBP. So, here’s another, more successful example of a journal compensating its reviewers.

And, in this case, what happened when they compensated reviewers? 1) more accepted reviews, 2) faster reviews, 3) no apparent decrease in quality or change in acceptance rates:

But, the downside to a journal – at Biology Open, the costs of running the program in 2025 totalled approximately ~£100,000, once reviewer and staff support costs were assessed.

Is this a sustainable financial model? The economics are certainly confronting, but it may ultimately be one part of the puzzle. There is probably a sweet spot of generative AI pre-processing and quality assurance to be combined with reviewer expertise and insight, and these sort of hybrid models will certainly be more widely seen.

I might be in the minority regarding appreciation of the unintended consequences of clinical IT, so I appreciate this call for a framework of formal documentation of the downstream impact of workflows changing as a result of CDS interventions. It’s quite easy to report sensitivity, specificity, AUROCs, NPVs, etc., but even clinically advantageous CDS can fail if the change in demand for services shifts the burden from traditional care pathways to new bottlenecks.

Their examples:

A good lens for technology developers to use when attempting to translate their innovation into the existing conditions and constraints of a health system.

A critique of YEARS has been whether it could be safely applied in patients with active cancer.

Your answer is “yes”:

For those of you counting at home, that’s 47 PEs diagnosed on the initial visit for the YEARS team and 57 PEs in the “CTPA everyone” team. In the 90-day follow-up window:

“In the YEARS group, 5 patients (1.8%) were diagnosed with VTE during the 3-month follow-up: 3 patients had a nonfatal PE and 2 patients had a possible PE-related death. In the CTPA-only group, 15 patients (5.5%) were diagnosed with VTE during the 3-month follow-up: 5 patients had a nonfatal PE, 2 patients had a fatal PE, 5 patients had a possible PE-related death, and 3 patients had a DVT.”

Clearly, the YEARS criteria does not have protective powers, but the result is reassuring nonetheless – and 22% fewer CTs performed to boot.

The study actually planned to enroll 1,566 patients to attain their initial statistical power for non-inferiority, but was able to stop early due to the preponderance of excess VTE in the CTPA-only group.

Go forth and risk-stratify!

The review process itself, however, is laborious – digesting clinical notes, lab results, and physiology, only to produce low frequencies of true missed opportunities. Sounds like the perfect job for an LLM!

And then:

As compared to a 2-physician review panel, the various LLMs – all a few months old, now – were able to extract varying percentages of “true positives” while including varying percentages of “false positives”. However, “true positives” are so infrequent the burden of “false positives” tips the scales most dramatically – and, as can be seen above, most cases flagged by the LLM are unlikely to represent missed opportunities for diagnosis. Sensitivity can be sacrificed for yield, so the low sensitivities seen here are not terribly concerning – but, even with poor sensitivity, the specificity remains inadequate.

It isn’t so much these LLMs are a “few months old” now, but that they are configured as one-shot prompts rather than a set of organized tool-using agents retrieving and organizing the data to best evaluate in the chosen framework. It might rather be surprising these LLMs performed as well as they did, considering the limited sophistication applied. Better work can be done!

But, in its place: fluids, fluids, fluids, and more fluids. Thus enshrined in Guidelines and sepsis protocols: fluids. This blunt instrument remained questioned, but untested, until CLOVERS – and now ARISE FLUIDS.

ARISE FLUIDS is an incremental step further with respect to fluid restriction. To be included, patients were enrolled in the emergency department and could not have received more than 1000mL of IVF as initial resuscitation. This allowed the total fluids curve to diverge more substantially:

And the outcomes were “similar”. The primary outcome was “days alive and out of hospital”, and this was not different between groups. There was a small point difference in 28 day mortality, but the 95% CI was 0.91 to 1.85 – imprecise enough to encompass a range of outcomes. You can see the overlapping survival curves develop a small difference early:

And the early – “non-significant” – difference remains durable throughout the follow-up period.

The most obvious challenge to interpreting these data from Australia and New Zealand is putting these outcomes in context next to sepsis care in other settings where typical fluid protocols far exceed even the “liberal” arm of this trial. Then, these data mostly only inform approaches to the “easy” patients who can be flexibly treated with either protocol – patients deemed “clinically unsuitable” (usually due to fluid restriction) were excluded, and these remain the types of patients for whom the question of fluids vs. vasopressors carries the greatest clinical significance.

It’s agents writ large, with an orchestration layer and an interface for fetching and orchestration, and the beautiful figure they’ve included gives you a lovely overview of the principles upon which it operates:

As you’ve probably gathered, this is basically a “full stack” clinician replacement, a simulation in a sandbox environment in which LLM-powered agents interact with a simulated patient and clinically generated data for independent diagnosis and management.

It’s all very much in silico, as the patients for the LLM clinicians are also role-played by LLMs, and these agents obviously behave in complete and fashionable ways unlike the true chaotic nature of an emergency department encounter. There’s an entire exploration using the MIMIC-IV information as ground-truth, and, like most of these demonstrations, they don’t get published unless they’re demonstrating human-equivalent performance.

Regardless, it’s a sneak preview of the endgame – copilots that run under the surface of EHRs acting as oversight, “guardian angels”, and, ultimately, potential clinician replacement allowing less-skilled healthcare workers to operate above their usual scope.

What if – however – it turned out the LLM wasn’t better?

What if it was no better than the obsequious Google AI result that comes with search?

That is the general gist of this brief communication, a trial of multiple frontier models against the current heavyweights of medical reference: OpenEvidence and UpToDate AI. Whether it’s MedQA (a), HealthBench (b), or Real Clinical Queries benchmark (c), the frontier models are on top:

Specifically, where they had real clinicians review the outputs from Real Clinical Queries benchmark, it is obvious the general models have it covered in all the pairwise comparisons:

And, naturally, any publication involving “frontier LLMs” – showcases performance of models already obsolete in present day!

Attempting to settle the debate with some finality, we have the BIHCA randomized trial for in-hospital cardiac arrest.

As seems to be the general theme for resuscitation trials, we find a “neutral” result:

A small absolute and relative difference in ROSC, a small absolute and (larger) relative difference in useful survival outcomes. There was little insight in the subgroup analysis as to which patients were feeding into these small differences:

Which, unfortunately, leaves us back where we started. The exclusion of sodium bicarbonate from resuscitation efforts is entirely valid. However, for those who have retained its use, there’s no obvious signal of harm, nor is bicarbonate expensive other than the time-cost and cognitive burden of its use during the acute event. The posterior Bayesian likelihoods of effectiveness are not strong enough to really move the needle on someone’s practice, one way or the other. And, so, another grand effort – which probably won’t change anything.

Well – they’re half-right.

In this primary care study where clinicians received the results of a BioFire rapid assay, clinicians were indeed less likely to prescribe antibiotics:

Sadly, however, when no virus was detected – they were *more likely* to prescribe antibiotics. The net overall effect of this viral testing, then, was: zero. Well, except for the added ~$400 cost of the assay.

The information gleaned by the panel provided no improvement in patient-oriented outcomes – regardless of the apparently “better targeted” antibiotic prescribing, both groups improved at the same rate:

Downstream “second sickening”, repeat prescribing, and other secondary outcomes were likewise no different.

This is further relevant, today, to emergency medicine – where ACEP has put up a draft clinical policy for feedback. This clinical policy, hoping to avoid unnecessary antibiotics and over-investigation in children, is proposing to recommend viral testing for all children under the age of 2 presenting with fever. There is very weak evidence to support this recommendation and, as you can see here, such testing is hardly a panacea for the underlying sociopsychobiomedical issues in the evaluation for infectious diseases.

But why not adulthood? And medicine?

This study report describes a role-playing game in which physicians received instruction regarding proper transfer of injured elderly patients to a specialized trauma center. The supplement provides a few screenshots, but, suffice to say, if you “undertriage” someone, you are a Bad Doctor:

These authors recruited 400 physicians to play their iPad game for at least 2 hours and compared them with 400 physicians who … well, they were just left to wallow in their ignorance. Following the exposure to the intervention, there was a difference in physician behaviors over the following year:

In the group exposed to game-based training, there was an increased likelihood of transfer for “severely injured” (AIS >16) elderly. Thus, the game wins – although, this doesn’t show the game is superior to another educational intervention, just against no education whatsoever.

Now, should we point out the lack of difference in patient-oriented outcomes? Is it reasonable to suggest clinicians in the “usual education” group simply made good decisions on their own clinical judgment, rather than a seeming obligation for transfer? These data aren’t nearly robust enough to support those assertions in any meaningful fashion, but it is an appropriate follow-up question for a future time.

Preschoolers are full of bugs – swabs find both common bacteria and common viruses. In patients with moderate-to-severe wheezing, then, in theory, could that inflammatory process be attenuated by clearing out the bacteria with our favorite indiscriminate antibiotic – azithromycin?

Nope.

It’s an interesting study – and does a nice job splitting their cohort into two chunks: two-thirds of wheezing preschoolers who tested positive for nasopharyngeal carriage of pathogenic bacteria, and a third of whom did not. Virtually all children also tested positive for viruses, predominantly rhinovirus. Approximately half of children enrolled were unwell enough to require brief hospitalization.

Regardless of bacteria “positive” or “negative” status, azithromycin administration did not change the course of symptom improvement when added to the rest of supportive care:

Ultimately, the study was stopped early for futility at interim analysis.

The authors also do note these data do not invalidate other data in which azithromycin was administered at the first sign of a wheezing illness – a different population whose inflammatory process has not fully manifested. But, for the rest of us – just put the antibiotics away!

But how much danger is out there, really?

This is just a retrospective data dredge of triage complaints implicating “atraumatic chest pain” from 141 hospitals staffed by the USACS group. From these 950,000 ED visits, the frequency of associated “life-threatening diagnoses” is as follows:

That is to say, if you see 20 patients a shift, work 15 shifts a month, and work for 20 years – you might see a handful of aortic dissections in your chest pain pile, and may never see the rarer ones.

Of course, there are plenty of limitations to these data – these only represent ED diagnoses, not final diagnoses following admission or further determination, nor do all these diagnoses present solely with “chest pain”. However, as a reasonable starting point in the ballpark of a baseline pretest likelihood of disease, these are fairly reasonable.

This is PhysicianBench – a suite of tasks created by researchers at Stanford requiring LLM-based clinical agents to execute sequential series of tasks mimicking those performed by clinicians in their daily work. This is a step forward beyond prior measures such as HealthBench, AgentClinic, EHRAgent, MedAgent, etc. in that it uses 1) a live EHR environment, 2) multiple chained tasks, 3) a framework for verification, 4) and based on tasks validated by clinicians in 21 specialties.

These look roughly like:

And the entire library is up on their GitHub.

There’s a pretty little graph:

Which shows GPT 5.5 was their top performer, completing approximately 46% of the tasks within their benchmarking library.

And then there’s so much fun in the paper – for example, are certain specialties more challenging than others?

And at what point in the tasks are the models crashing out?

Now, there can always be quirks in the construction of the test environment putting the thumb on certain models versus others, but the results have gross face validity. Then, these are not “physician replacement”, they are just taskrabbits – and these are general models, not versions designed to execute clinical tasks. So, we’re not quite at reliable problem-solving minion level, but likely only a few generations away.

It’s a bit of a complex trial with all manner of risk of bias embedded within. The basic premise, however, is that Google recruited folks within the Fitbit application to participate – at which point they were able to interact with a “symptom checker” app to hold a conversation with possible diagnoses suggested at the end:

There were five different chatbot architectures, roughly: 1) a basic Gemini-style general response engine, 2) a clinical health agent adherent to a fixed set of clinical questions, 3) the same clinical health agent with a flexible approach to a set of clinical questions, 4) a health agent that dynamically created ranked diagnoses and asked follow-up clarifying questions, and 5) a health agent with unstructured clinical evaluation. Nearly 14,000 participants chatted with these agents, but the study population comes from the 1,228 who reported a subsequent ground-truth clinical diagnosis for their symptoms from any sort of follow-up. Then, 517 of these were selected for clinical review as their final cohort of analysis.

Part of the evaluation here involved comparing the accuracy of the diagnoses generated by these various interaction strategies – and, interestingly, only the basic Gemini version was clearly inferior to the rest, all of which performed similarly.

Then, there was a phase in which two clinicians read the transcripts in the study population and provided their own “top five” list of final diagnoses. A third clinician then adjudicated the three lists of differential diagnoses – two clinicians and SymptomAI – with SymptomAI’s list being the most commonly chosen as “top ranked”. Then, compared to the final self-reported ground truth diagnoses, SymptomAI was slightly better:

There’s also a further analysis in which the reported symptoms and diagnoses were linked to Fitbit data, and mild inflammatory physiology was linked to influenza-like illness and the like.

So, some useful tidbits absent any dramatic insights. It’s interesting to see the different conversational prompting strategies produced similar results. The data linking SymptomAI findings to physiologic phenotypes is non-specific, but valid nonetheless. The part where SymptomAI “outperforms” clinicians is a little more precarious. The validity of this analysis hinges on the self-reported diagnoses of a subset of participants, subject to a spectrum of reporting biases. Then, the clinicians are hamstrung by making their diagnoses off a transcript and line of inquiry divergent from their own approach. And, as seen in the figures above, Top-5 accuracy was still only ~70%, meaning there’s more work to be done integrating these data into subsequent diagnostic and management plan.

Not quite ready for prime time, but, every day, the science marches on!

From George Clooney in ER to Noah Wyle in The Pitt, emergency department doctors have long been popular heroes. But will it soon be time to hang up the scrubs?

A groundbreaking Harvard study has found that AI systems outperformed human doctors in high-pressure emergency medicine triage, diagnosing more accurately in the potentially life and death moments when people are first rushed to hospital.

And here’s the proof:

Each study “participant” performed three different generations of five potential diagnoses, one at each stage of information availability. These stages were the clinical content available as generated by nurse triage, after an ED physician had documented the encounter, and then upon admission to the hospital. Additional physician members of the study team scored each list of possible diagnoses as the percentage having exact or near-exact correct diagnosis (“Bond Score”). Purple bar (o1-preview) wins!

Of course, 1) these are what internal medicine clinicians made out of emergency department clinical data, 2) the “exact diagnosis” is the not the intent of ED information gathering, it is exclusion of immediately sinister conditions and early treatment for prevention of ongoing morbidity/mortality, 3) the entire foundation of this demonstration is the information produced by the expert evaluation, management, and cognitive work of human emergency physicians, and 4) sure, neat trick, but another contrived experiment in silico, not in Real Life.

With predictable findings, they’ve also fed o1-preview NEJM Healer cases, Grey Matters cases, and Clinicopathological Conferences and compared them with previous 4o results. Another generation of models, another incremental improvement in performance:

It also ought to be noted – o1 is last year’s OpenAI flagship, whereas we’re well beyond that today. OpenAI o1 is also just a “general” LLM, rather than one specifically tuned or trained for optimal performance on clinical content.

As for the calls for sunsetting of clinical autonomy, we’re not quite yet there – but the transformation is well in progress. Even if there are ways to nitpick these results, the frog is already grasping at our heels.

Those days are assuredly over – and likely have been for some time.

This is a publication of the results of PANORAMIC and CanTreatCOVID trials, two trials conducted in the U.K. and Canada in the 2021 to 2024 timeframe. We’ve already seen Paxlovid has no value in the “standard risk” population – and this revisits its utility in a higher-risk population, but a more modern cohort than those enrolled in EPIC-HR back in 2021.

The research summary infographic from NEJM probably summarizes this the best:

The big difference seen here – in EPIC-HR, 7% of the placebo cohort suffered hospitalization or death within 28 days, as opposed to the ~1% in these trials.

Nirmatrelvir-ritonavir definitely reduces viral load faster than placebo, and about 1 in 10 reported symptom improvement sooner as a result. Adverse events were reported frequently with nirmatrelvir-ritonavir, but very few were classified as serious.

Toss it on the hill with the influenza antivirals: in the ambulatory population, no clear benefit in serious outcomes, probably minimal benefit for symptomatic recovery.

The preponderance of evidence now, in adults and children, is that plain amoxicillin ought to be the first choice:

Out of ~500,000 eligible encounters, ~400,000 received amoxicillin-clavulanic acid, and the results above show propensity score-matched cohorts. No reliable detectable differences in treatment failure, but amoxicillin-clavulanic acid gave rise to more downstream cases of yeast infections and Clostridium difficile.

But, not mentioned at all in this article: it’s likely most of these cases never needed anything more than time and conservative therapy. A clear reason it’s not possible to show a difference in outcomes between two therapies is when the natural course of the disease is to cure, regardless. Without stricter criteria for failure of conservative management and initiation of antibiotics in sinusitis, we can’t truly ask the question whether there is a difference in efficacy between these two agents.

At the least, however, if we can’t stop prescribing unnecessary antibiotics – prescribe the less harmful one.