Hopeless Sepsis Criteria, Pediatric Edition

Academic criteria trying to mutate into clinical application.

If you’ve been dismayed by the ongoing infatuation of CMS with SEP-3, the sepsis “quality” measure for adults, I’m sure you’ll be excited to hear about the present obsession with the Phoenix Sepsis Criteria for children.

Children with sepsis, just like in adults, see high mortality once organ dysfunction sets in. Naturally, therefore, early recognition and appropriate escalation are necessary. The PSC, however, are useful for objectively defining the presence of sepsis, but, by that point, the ship has sailed as far as any missed opportunities for early intervention. After all, the PSC includes such rarely-collected lab values as INR, fibrinogen and D-dimer, and includes features such as respiratory and circulatory support.

In this study, the PECARN family gets together to try and use machine learning methods to detect which children presenting to the emergency department will go on to develop PSC-defined sepsis and septic shock. With 1.6M visits to train (0.35% sepsis, 0.15% septic shock) and 700k held out to test, these data scientists were able to develop a model with AUROC of 0.936. This sounds OK until the rubber hits the road choosing a threshold for clinical implementation – specificities of ~0.8 when mandating a sensitivity of 0.9:

As you can see, this means the positive predictive value of a positive result is ~2% – 50 false positives for every true positive. The utility of their model is also further limited as seen in their SHAP values, where ESI triage criteria are the most important features – and performance degrades when this human insight is removed from the model.

So, an attempt to develop a gateway to prognosticating high-risk PSC children in the ED – sadly, and predictably, unusable.