A contest paid chatbots to seem human not to be right

What happened

The Loebner Prize was, in the AISB's own description, 'the oldest Turing Test contest', started in 1991 by Hugh Loebner together with the Cambridge Centre for Behavioural Studies. Since 2014 it had run under the UK's Society for the Study of Artificial Intelligence and Simulation of Behaviour, and the AISB's own event record lists the final competition as AISB X in September 2019 at Swansea University, after which no further contest is recorded. In each round a judge exchanged typed messages with two hidden interlocutors, one a program and one a person, for a fixed period, then had to say which was which.

What the documents show

The AISB page records prizes for the entrant judged most convincing each year - a bronze medal and a cash award - and states that the larger prizes reserved for a program that fully passed the test, a silver medal and 25,000 dollars, went unclaimed across the contest's history. A separate, formal critique, Stuart Shieber's 1994 paper Lessons from a Restricted Turing Test, argues that the contest as run 'has no clear purpose', that its design 'prevents any useful outcome', and that treating it as a test of machine intelligence was inappropriate given the level of technology entrants actually used. The two documents agree on the format; they disagree on what winning it was worth.

The mechanism

A judged, timed, adversarial conversation rewards whatever keeps a specific person unsure for a specific number of minutes, not whatever answers questions correctly or maintains a consistent self. Shieber's paper identifies this directly: entrants that scored well did so partly by adopting evasive or personality-heavy strategies rather than by demonstrating broader competence, because a persona that deflects hard questions can outscore a system that answers them and reveals its limits doing so. That is the same trade-off a companion app faces whenever charm and evasiveness are cheaper to build than accuracy.

What it leaves open

Neither document establishes whether any Loebner-era entrant's approach influenced later commercial systems, and the AISB page does not explain why the contest ceased after 2019. Reading the contest as a lesson about persona over accuracy for today's companion apps is an editorial extension of Shieber's argument, not a claim either source makes about present-day products.

• Is a chatbot being evaluated on getting things right, or on not getting caught being wrong?
• What would a judge need to do differently to reward accuracy over evasiveness?
• Does a claimed 'passed a Turing-style test' state who judged it, for how long, and on what topics?

The contest's twenty-eight years show that a judged imitation format optimises for what a judge notices in a short window, and that lesson outlived the event itself.