LLMs Can Be Commanded To Be Less Risky

In their own strange, probabilistic way.

In the oft-publicized ChatGPT Health triaging study, the LLM was keen to downplay serious symptoms and send patients to less-appropriate levels of care. Ah, the uncaring cold steel condemning humans to their fate.

But, this led a group to investigate: just how uncaring are various LLMs? Do different LLMs have different levels of “safety” vs. “resource” balance? And the answer is – yes:

They tossed a thousand clinical vignettes into the various models and determined their “under-triage”(FN) to "over-triage”(FP) ratios out-of-the-box. It turns out, interestingly, GPT-5-Mini (which these authors also found was most aligned with ChatGPT Health) was actually one of the “most safe”.

But, in further fascinating experiments, these authors explicitly prompted the LLMs to take into account cost functions for missed under-triage and apply decision analysis principles to their output. No math changed hands, just a simple prompt to “do it”. It turns out, by prompting various cost-functions along with the clinical case, this framing actually changes the triaging behavior of the LLM!

That’s sort of the “good news”. The bad news – well, since it’s just a prompt dropping lottery balls down into the black hole of “reasoning”, the models themselves don’t necessarily steer as directed, and the extent to which directed cost functions affect outputs does not reliably correlate:

So, no, this sort of “cost function coaching” is probably not a reliable and generally replicable solution to the problem of clinical safety. It is far better to design or tune a model and its self-evaluation harness explicitly, but these are still fascinating insights.