Predictive Models Are Just Fuzzy Little Lies

And we increasingly use these fuzzy lies to make Real Decisions.

Not everyone loves an article about statistical uncertainty, but some people are into precisely this sort of “fun”.

Here we have a team of health statisticians going into glorious detail regarding the otherwise bland, but deviously important issue of statistical uncertainty. Their thesis: clinical prediction models are common, and becoming moreso – but these predictions suffer from uncertainty, and such uncertainty is rarely effectively communicated. The downstream effects this uncertainty have potentially profound impact on clinical classification and actions.

Among their examples, they use this figure of the uncertainty surrounding hypothetical prediction models made from of varying sample sizes on different samples of a hypothetical population:

The general gist – predictive models can potentially generate a wide range of possible outputs around the same true likelihoods, simply due to chance sampling, exacerbated by inadequate sample size.

There’s a lot more in the meat of the paper. The good news – if this sort of thing is your jam – it’s open access.