The Turing Test was proposed by British mathematician Alan Turing in 1950 as a way of assessing whether a machine could demonstrate human-like intelligence. In its best-known form, a person communicates with both a human and a machine without knowing which is which and tries to identify the machine.
In a new study, Te Whare Wānanga o Waitaha | University of Canterbury (UC) philosopher Professor Diane Proudfoot joins computer scientist PhD student Sharon Temtsin and Professor Christoph Bartneck to argue that many experiments described as “Turing Tests” do not follow the test Turing originally proposed.
The researchers say they have implemented Turing’s original conceptual design experimentally for the first time.
The team examined Turing’s writings from 1948 to 1952, along with early commentary and historical sources, to determine how the imitation game was intended to work.
“Features commonly associated with the Turing Test, including a five-minute time limit and the idea that a machine passes if it fools 30% of judges, were not intended by Turing as rules for passing the test,” Professor Bartneck explains.
“Rather, Turing envisaged a three-player imitation game involving an interrogator, a human and a machine, with open-ended questioning and no fixed time limit. This significantly changes the goalposts and therefore the results for this test.”
To put their interpretation into practice, the researchers ran an experiment using GPT-4-Turbo, an AI language model developed by OpenAI, the company behind ChatGPT.
Participants conversed freely with both a human and the AI system until they felt confident enough to decide which was the machine.
Of 37 participants, 36 correctly identified GPT-4-Turbo – a recognition rate of 97%.
The study also found that 33 participants, or 89%, took longer than five minutes to make their decision. The average conversation lasted 14 minutes, while the longest lasted 46 minutes.
The research raises questions about whether short-duration tests give people enough time to meaningfully probe an AI system.
“Turing’s open-ended format gives interrogators an opportunity to test whether a system genuinely understands a topic, rather than simply producing convincing responses,” Professor Bartneck says.
“When this is taken into account, we’d argue that no large language model has yet been shown to pass a test that follows the design in Turing’s original work.”
The researchers stress that their findings do not show that all current or future AI models would fail. Their experiment used GPT-4-Turbo, and newer models may perform differently.
The study, Why No-One Has Implemented the Turing Test: Identifying Turing’s Design and Putting It Into Practice, published in Philosophy of Science, calls for the experiment to be repeated using different AI models, prompts and participants.
