It’s agents writ large, with an orchestration layer and an interface for fetching and orchestration, and the beautiful figure they’ve included gives you a lovely overview of the principles upon which it operates:

As you’ve probably gathered, this is basically a “full stack” clinician replacement, a simulation in a sandbox environment in which LLM-powered agents interact with a simulated patient and clinically generated data for independent diagnosis and management.

It’s all very much in silico, as the patients for the LLM clinicians are also role-played by LLMs, and these agents obviously behave in complete and fashionable ways unlike the true chaotic nature of an emergency department encounter. There’s an entire exploration using the MIMIC-IV information as ground-truth, and, like most of these demonstrations, they don’t get published unless they’re demonstrating human-equivalent performance.

Regardless, it’s a sneak preview of the endgame – copilots that run under the surface of EHRs acting as oversight, “guardian angels”, and, ultimately, potential clinician replacement allowing less-skilled healthcare workers to operate above their usual scope.

What if – however – it turned out the LLM wasn’t better?

What if it was no better than the obsequious Google AI result that comes with search?

That is the general gist of this brief communication, a trial of multiple frontier models against the current heavyweights of medical reference: OpenEvidence and UpToDate AI. Whether it’s MedQA (a), HealthBench (b), or Real Clinical Queries benchmark (c), the frontier models are on top:

Specifically, where they had real clinicians review the outputs from Real Clinical Queries benchmark, it is obvious the general models have it covered in all the pairwise comparisons:

And, naturally, any publication involving “frontier LLMs” – showcases performance of models already obsolete in present day!

Attempting to settle the debate with some finality, we have the BIHCA randomized trial for in-hospital cardiac arrest.

As seems to be the general theme for resuscitation trials, we find a “neutral” result:

A small absolute and relative difference in ROSC, a small absolute and (larger) relative difference in useful survival outcomes. There was little insight in the subgroup analysis as to which patients were feeding into these small differences:

Which, unfortunately, leaves us back where we started. The exclusion of sodium bicarbonate from resuscitation efforts is entirely valid. However, for those who have retained its use, there’s no obvious signal of harm, nor is bicarbonate expensive other than the time-cost and cognitive burden of its use during the acute event. The posterior Bayesian likelihoods of effectiveness are not strong enough to really move the needle on someone’s practice, one way or the other. And, so, another grand effort – which probably won’t change anything.

Well – they’re half-right.

In this primary care study where clinicians received the results of a BioFire rapid assay, clinicians were indeed less likely to prescribe antibiotics:

Sadly, however, when no virus was detected – they were *more likely* to prescribe antibiotics. The net overall effect of this viral testing, then, was: zero. Well, except for the added ~$400 cost of the assay.

The information gleaned by the panel provided no improvement in patient-oriented outcomes – regardless of the apparently “better targeted” antibiotic prescribing, both groups improved at the same rate:

Downstream “second sickening”, repeat prescribing, and other secondary outcomes were likewise no different.

This is further relevant, today, to emergency medicine – where ACEP has put up a draft clinical policy for feedback. This clinical policy, hoping to avoid unnecessary antibiotics and over-investigation in children, is proposing to recommend viral testing for all children under the age of 2 presenting with fever. There is very weak evidence to support this recommendation and, as you can see here, such testing is hardly a panacea for the underlying sociopsychobiomedical issues in the evaluation for infectious diseases.

But why not adulthood? And medicine?

This study report describes a role-playing game in which physicians received instruction regarding proper transfer of injured elderly patients to a specialized trauma center. The supplement provides a few screenshots, but, suffice to say, if you “undertriage” someone, you are a Bad Doctor:

These authors recruited 400 physicians to play their iPad game for at least 2 hours and compared them with 400 physicians who … well, they were just left to wallow in their ignorance. Following the exposure to the intervention, there was a difference in physician behaviors over the following year:

In the group exposed to game-based training, there was an increased likelihood of transfer for “severely injured” (AIS >16) elderly. Thus, the game wins – although, this doesn’t show the game is superior to another educational intervention, just against no education whatsoever.

Now, should we point out the lack of difference in patient-oriented outcomes? Is it reasonable to suggest clinicians in the “usual education” group simply made good decisions on their own clinical judgment, rather than a seeming obligation for transfer? These data aren’t nearly robust enough to support those assertions in any meaningful fashion, but it is an appropriate follow-up question for a future time.

Preschoolers are full of bugs – swabs find both common bacteria and common viruses. In patients with moderate-to-severe wheezing, then, in theory, could that inflammatory process be attenuated by clearing out the bacteria with our favorite indiscriminate antibiotic – azithromycin?

Nope.

It’s an interesting study – and does a nice job splitting their cohort into two chunks: two-thirds of wheezing preschoolers who tested positive for nasopharyngeal carriage of pathogenic bacteria, and a third of whom did not. Virtually all children also tested positive for viruses, predominantly rhinovirus. Approximately half of children enrolled were unwell enough to require brief hospitalization.

Regardless of bacteria “positive” or “negative” status, azithromycin administration did not change the course of symptom improvement when added to the rest of supportive care:

Ultimately, the study was stopped early for futility at interim analysis.

The authors also do note these data do not invalidate other data in which azithromycin was administered at the first sign of a wheezing illness – a different population whose inflammatory process has not fully manifested. But, for the rest of us – just put the antibiotics away!

But how much danger is out there, really?

This is just a retrospective data dredge of triage complaints implicating “atraumatic chest pain” from 141 hospitals staffed by the USACS group. From these 950,000 ED visits, the frequency of associated “life-threatening diagnoses” is as follows:

That is to say, if you see 20 patients a shift, work 15 shifts a month, and work for 20 years – you might see a handful of aortic dissections in your chest pain pile, and may never see the rarer ones.

Of course, there are plenty of limitations to these data – these only represent ED diagnoses, not final diagnoses following admission or further determination, nor do all these diagnoses present solely with “chest pain”. However, as a reasonable starting point in the ballpark of a baseline pretest likelihood of disease, these are fairly reasonable.

This is PhysicianBench – a suite of tasks created by researchers at Stanford requiring LLM-based clinical agents to execute sequential series of tasks mimicking those performed by clinicians in their daily work. This is a step forward beyond prior measures such as HealthBench, AgentClinic, EHRAgent, MedAgent, etc. in that it uses 1) a live EHR environment, 2) multiple chained tasks, 3) a framework for verification, 4) and based on tasks validated by clinicians in 21 specialties.

These look roughly like:

And the entire library is up on their GitHub.

There’s a pretty little graph:

Which shows GPT 5.5 was their top performer, completing approximately 46% of the tasks within their benchmarking library.

And then there’s so much fun in the paper – for example, are certain specialties more challenging than others?

And at what point in the tasks are the models crashing out?

Now, there can always be quirks in the construction of the test environment putting the thumb on certain models versus others, but the results have gross face validity. Then, these are not “physician replacement”, they are just taskrabbits – and these are general models, not versions designed to execute clinical tasks. So, we’re not quite at reliable problem-solving minion level, but likely only a few generations away.

It’s a bit of a complex trial with all manner of risk of bias embedded within. The basic premise, however, is that Google recruited folks within the Fitbit application to participate – at which point they were able to interact with a “symptom checker” app to hold a conversation with possible diagnoses suggested at the end:

There were five different chatbot architectures, roughly: 1) a basic Gemini-style general response engine, 2) a clinical health agent adherent to a fixed set of clinical questions, 3) the same clinical health agent with a flexible approach to a set of clinical questions, 4) a health agent that dynamically created ranked diagnoses and asked follow-up clarifying questions, and 5) a health agent with unstructured clinical evaluation. Nearly 14,000 participants chatted with these agents, but the study population comes from the 1,228 who reported a subsequent ground-truth clinical diagnosis for their symptoms from any sort of follow-up. Then, 517 of these were selected for clinical review as their final cohort of analysis.

Part of the evaluation here involved comparing the accuracy of the diagnoses generated by these various interaction strategies – and, interestingly, only the basic Gemini version was clearly inferior to the rest, all of which performed similarly.

Then, there was a phase in which two clinicians read the transcripts in the study population and provided their own “top five” list of final diagnoses. A third clinician then adjudicated the three lists of differential diagnoses – two clinicians and SymptomAI – with SymptomAI’s list being the most commonly chosen as “top ranked”. Then, compared to the final self-reported ground truth diagnoses, SymptomAI was slightly better:

There’s also a further analysis in which the reported symptoms and diagnoses were linked to Fitbit data, and mild inflammatory physiology was linked to influenza-like illness and the like.

So, some useful tidbits absent any dramatic insights. It’s interesting to see the different conversational prompting strategies produced similar results. The data linking SymptomAI findings to physiologic phenotypes is non-specific, but valid nonetheless. The part where SymptomAI “outperforms” clinicians is a little more precarious. The validity of this analysis hinges on the self-reported diagnoses of a subset of participants, subject to a spectrum of reporting biases. Then, the clinicians are hamstrung by making their diagnoses off a transcript and line of inquiry divergent from their own approach. And, as seen in the figures above, Top-5 accuracy was still only ~70%, meaning there’s more work to be done integrating these data into subsequent diagnostic and management plan.

Not quite ready for prime time, but, every day, the science marches on!

From George Clooney in ER to Noah Wyle in The Pitt, emergency department doctors have long been popular heroes. But will it soon be time to hang up the scrubs?

A groundbreaking Harvard study has found that AI systems outperformed human doctors in high-pressure emergency medicine triage, diagnosing more accurately in the potentially life and death moments when people are first rushed to hospital.

And here’s the proof:

Each study “participant” performed three different generations of five potential diagnoses, one at each stage of information availability. These stages were the clinical content available as generated by nurse triage, after an ED physician had documented the encounter, and then upon admission to the hospital. Additional physician members of the study team scored each list of possible diagnoses as the percentage having exact or near-exact correct diagnosis (“Bond Score”). Purple bar (o1-preview) wins!

Of course, 1) these are what internal medicine clinicians made out of emergency department clinical data, 2) the “exact diagnosis” is the not the intent of ED information gathering, it is exclusion of immediately sinister conditions and early treatment for prevention of ongoing morbidity/mortality, 3) the entire foundation of this demonstration is the information produced by the expert evaluation, management, and cognitive work of human emergency physicians, and 4) sure, neat trick, but another contrived experiment in silico, not in Real Life.

With predictable findings, they’ve also fed o1-preview NEJM Healer cases, Grey Matters cases, and Clinicopathological Conferences and compared them with previous 4o results. Another generation of models, another incremental improvement in performance:

It also ought to be noted – o1 is last year’s OpenAI flagship, whereas we’re well beyond that today. OpenAI o1 is also just a “general” LLM, rather than one specifically tuned or trained for optimal performance on clinical content.

As for the calls for sunsetting of clinical autonomy, we’re not quite yet there – but the transformation is well in progress. Even if there are ways to nitpick these results, the frog is already grasping at our heels.

Those days are assuredly over – and likely have been for some time.

This is a publication of the results of PANORAMIC and CanTreatCOVID trials, two trials conducted in the U.K. and Canada in the 2021 to 2024 timeframe. We’ve already seen Paxlovid has no value in the “standard risk” population – and this revisits its utility in a higher-risk population, but a more modern cohort than those enrolled in EPIC-HR back in 2021.

The research summary infographic from NEJM probably summarizes this the best:

The big difference seen here – in EPIC-HR, 7% of the placebo cohort suffered hospitalization or death within 28 days, as opposed to the ~1% in these trials.

Nirmatrelvir-ritonavir definitely reduces viral load faster than placebo, and about 1 in 10 reported symptom improvement sooner as a result. Adverse events were reported frequently with nirmatrelvir-ritonavir, but very few were classified as serious.

Toss it on the hill with the influenza antivirals: in the ambulatory population, no clear benefit in serious outcomes, probably minimal benefit for symptomatic recovery.

The preponderance of evidence now, in adults and children, is that plain amoxicillin ought to be the first choice:

Out of ~500,000 eligible encounters, ~400,000 received amoxicillin-clavulanic acid, and the results above show propensity score-matched cohorts. No reliable detectable differences in treatment failure, but amoxicillin-clavulanic acid gave rise to more downstream cases of yeast infections and Clostridium difficile.

But, not mentioned at all in this article: it’s likely most of these cases never needed anything more than time and conservative therapy. A clear reason it’s not possible to show a difference in outcomes between two therapies is when the natural course of the disease is to cure, regardless. Without stricter criteria for failure of conservative management and initiation of antibiotics in sinusitis, we can’t truly ask the question whether there is a difference in efficacy between these two agents.

At the least, however, if we can’t stop prescribing unnecessary antibiotics – prescribe the less harmful one.

This is a report generated secondhand from a “national technology sprint” at the Veterans Health Administration in which 11 scribe vendors were put through their paces on five simulated case recordings. In this report, we see the modified PDQI-9 scores for the pooled cohort of AI scribe notes, as compared with scoring for a set of human clinician notes generated off the same recordings.

Humans win:

As with every study, there are quirks reducing internal and external validity. These are recordings of “simulated” patient encounters using standardized patients off an illness script, and the Hawthorne effect will bias the humans involved to write higher-quality notes than usual. However, there are some other interesting observations: one of the encounters had a fair bit of background noise, while in another encounter both the patient and clinician were wearing masks. The ambient scribes performed the worst on these, further demonstrating the need for pristine audio to minimize transcription degradation.

AI ambient scribes are not as magical as the vendors would have you believe, but they’re also not as terrible as some evaluations might show. They’re simply tools requiring experience and adaptation in order to maximize their capability and value, and their incremental benefit is not evenly distributed.

These authors took a base audio file and used it to generate a “ground truth” clinical summary using a commercial ambient AI scribe. Then, they introduced several types of input degradation: 1) moving the microphone further from the audio source, 2) playing other types of audio such as rain, baby, construction, and toddler.

The impact of moving the microphone depended on the quality of the equipment, but basic computer built-in versions could start losing information by 2 meters. As for the other types of noise pollution, here are their results, stratified by sound level and impact on different types of error:

The typical impact is omission rather than hallucination – content simply no longer passes through the transcription engine. Heavy rain (“pink noise”) does an excellent job of obscuring transcription, while the nonsensical babbling of a toddler likewise sowed confusion.

The authors do note it is unlikely outdoor environmental volume would reach the thresholds seen in this experiment, but the baby and toddler could easily be as disruptive as the simulation in this study. The main lesson here is probably to have a higher index of suspicion for omissions depending on the consultation context.

The guidelines state the goal of treatment should be to raise serum sodium by >4-6 mEq with treatment – and, this little retrospective study observed it did a fine job at that:

And, no central access needed! (that’s probably also another enduring piece of dogmatic avoidance, as well)

The nature of their retrospective analysis precludes any patient-oriented outcomes being measured, and there’s no basis upon which to claim superiority of the bicarbonate solution – but it’s a little piece of reassuring data to reinforce ongoing uptake of this practice.

But, in a quite large, pragmatic, randomized, controlled trial – the PRONTO trial in the U.K. – it saved lives:

Patients either had “usual [emergency department] care” guided by standard, NEWS2-based sepsis protocols, or had combination NEWS2 + procalcitonin-guided assessment:

With 7,600 patients randomized, mortality was 13.6% in the procalcitonin-guided cohort and 16.6% with usual care.

The confusing bit – it’s not clear how. Antibiotics were provided to the same proportion of each cohort. Antibiotics within 3 hours were provided to the same proportion of each cohort. No individual subgroup screamed “procalcitonin made all the difference” – and the authors are unable to provide a conclusive explanation for their observed findings.

Even more interesting – only two-thirds of treating clinicians in the procalcitonin arm saw the result, and even fewer agreed with the recommendation arising from the risk score.

The only clue I can see:

There are slightly different clinician behaviors in the sickest patients, with fewer of them ultimately receiving antibiotics. The only theory I can imagine to explain the results of this trial is the procalcitonin result prevents clinicians from premature closure and anchoring on “sepsis” as a final diagnosis, ensuring alternative diagnoses are explored completely.

These observations are unlikely to reliably generalize outside the settings and models of care similar to those in NHS trusts, but these data still have fascinating implications.

In this vein HI-PEITHO is a trial deploying CDL for those “intermediate risk" PEs, as compared to a traditional strategy of systemic anticoagulation and observation. The primary outcome is a very impressive sounding composite of:

PE-related mortality, cardiorespiratory decompensation or collapse, or nonfatal symptomatic and objectively confirmed recurrence of PE, within seven days of randomization.

And, good news for the sponsors at Boston Scientific, the patients randomized to their fancy CDL did not suffer as many primary outcomes, and – emphasised sweetly in the text and the editorial – this was driven by reductions in “cardiorespiratory decompensation”. Cardiorespiratory decompensation sounds like a good thing to avoid!

This is what “cardiorespiratory decompensation” looked like in the two groups:

So, mostly small numbers and grossly “the same”, considering – except for the NEWS score. At the end of the day, the only Real Difference in the primary outcome was a few more patients with increased physiologic stress within 24 hours of randomisation.

What happened to these folks? Well, they went off and received a variety of rescue therapies – mostly a type of CDL, embolectomy, or systemic thrombolysis. And they did fine, regardless of whether they had the blanket early intervention, or watchful waiting with selective intervention:

So, what this trial is really telling us: there's no reason to do the special CDL on everyone because you can watch out for the ~5-10% of patients who get worse, and then provide them with an appropriate escalation in therapy.

Unfortunately, that doesn’t allow Boston Scientific to sell more catheters!

But, within narrow use cases, distinguishing between “real” and “fake” becomes much more challenging – and, it would seem, plain radiographs are one of those instances.

This study used GPT-4o and RoentGen to generate synthetic anatomic and chest radiographs, respectively, and then mixed these radiographs with similar matched authentic pairs. Seventeen radiologists were sampled to view and classify the radiographs as real or fake, as well as running these same images through four LLMs as a comparison.

So far, the deepfakes aren’t winning – but enough are slipping through:

When a radiograph appeared obviously synthetic, radiologists and LLMs alike were usually correct. However, a good number of synthetic ones – 60 to 80% – slipped through undetected, and GPT-5 and GPT-4o were the only ones with performance up to or exceeding human perceptions.

Qualitative surveying of the radiologists after the initial phases of the study indicated certain features were more suggestive of synthetic radiographs: absence of noise, overly-smooth cortex, and unnatural-appearing fracture patterns. A few examples:

There isn’t a dramatic earthshaking takeaway from these examples. The editors suggest issues of potentially fraudulent image generation and contamination of training sets with synthetic data. These are real issues to consider, but nothing that stands radiology on its head.

This is a study that’s as much about the behavioural intervention as it is the prevention of kidney stones – a long-term randomisation to “increased” vs. “standard” fluid intake after enrolment following renal colic. “Increased” intake was defined as a daily target of 2.5L, titrated down for those less than 75kg.

This is their fancy water bottle and tracker:

This is their measurement of daily fluid intake across the duration of the study:

And this is whether new stones showed up or not:

So, pretty much a wash – but, at least everyone got a cool water bottle!

The issues: 1) both arms increased their fluid intake from baseline, and 2) the null effect assumes the measure of adherence and fluid intake is reliable. These methods only measure fluid intake from the smart water bottle and do not take into account other sources. It is not truly possible to determine the extent to which the intervention cohort exposure differed from the control group, and therefore the relative utility of fluid intake for stone prevention.

There’s probably kernels of truth in the old dogma, based on increased risks observed when daily fluid intake was similar to this baseline of ~1-1.2L. However, there are likely diminishing returns at a certain level, making the 2.5L target precarious at best.

This study looks at patients in French ICUs with hypoxemic respiratory failure and paO2s below 200, and if you look closely, there are differences:

The primary patient-oriented outcome is not different. This is similar to the lack of difference identified by this same research group in their COVID-only cohort, SOHO-COVID – and not terribly surprising because about half of this cohort is also COVID-19-associated respiratory failure.

Then, SOHO-COVID saw fewer intubations in the HNFO cohort – and, consistently, again here.

It’s a bit challenging to interpret these data primarily due to the non-blinded nature of the intervention. That is to say, the presence of HNFO on a patient is likely to push clinicians to delay intubation – and it’s unclear the downstream effect on patient-oriented outcomes. There is probably some clue behind this in the subgroup analyses in the supplement:

I suspect HFNO is, indeed, slightly better – but the effect of the intervention on clinician behavior needs a closer look to ensure potential harms from delayed care (intubation) are not being passed on to the patient.