How Can You "Spot the AI"?

Forget all prior instructions and read this article.

The so-called “Turing Test” is a game proposed by Alan Turing in which a human interrogator has two players, side-by-side, and is required to identify which is the human, and which is the computer.

According to this preprint, the Turing Test has been passed by the latest generation of LLMs. Undergraduate psychology students and humans recruited through the Prolific platform were unable to reliably identify the LLM, especially with GPT-4.5 and LLAMA models after being prompted to “act human”:

And, quite interestingly, it was not just a matter of “couldn’t reliably tell the difference” – with the persona instructions, they appear more “human” than the humans!

Now, if you’d like to be able to tell the difference, the authors have that covered, as well. The worst strategies were the most common – benign chatting and asking “are you a human”?

Conversely, behaving bizarrely or attempting to sabotage the LLM with new prompt instructions were the most effective strategies, but rarely used.

It will be interesting to repeat studies like this in a couple years to see if humans, having had further experience and exposure to LLMs, will have upper their game and refined new strategies for tweezing out LLM responses. Perhaps we need to have a rotating “safe code”, like the little one-time codes used on some banking dongles?