Tuesday 10 June 2014

Eugene Goostman machine convinced 33.33% of a new set of Judges at The Royal Society 6-7 June 2014, following its 29.17% performance at Bletchley Park in 2012

Eugene Goostman has done it again! *No, it has bettered what it did previously: the machine entry convinced one third of 30 Judges that it was human in Turing tests carried out at The Royal Society London, 6-7 June 2014.

The Turing test event was independently adjudicated by Professor John Barnden of Birmingham University, supported by RoboLaw project's Dr. Fiorella Battaglia and Dr. Federica Lucivero.

For the record, at The Royal Society London  Vladimir Veselov, Eugene Demchenko and Sergey Ullasen, the team behind machine entry Eugene Goostman, surpassed their previous record, convincing 25% of the Judges, at Reading University in 2008 increasing that to convincing 29.17% of the Judges at Bletchley Park in 2012 that Eugene was a human - see the New Scientist report by Celeste Biever, one of the Judges in 2012 here:

"Bot with boyish personality wins biggest Turing" test: http://www.newscientist.com/blogs/onepercent/2012/06/bot-with-boyish-personality-wi.html

[NB: IBM's Watson team headed by David Ferrucci declined to take part in Turing100 at Bletchley Park in 2012. Watson is a reverse question-answer deep-search system and not built for making human conversation.]

For clarification, in 2012 two versions of Turing's Imitation Game were implemented:

1) The simultaneous comparison Turing test in which a human Judge interrogated two hidden entities at the same time

2) The viva voce Turing test in which the human Judge interrogated one hidden entity at the same time

The purpose of the 2012 Turing test experiment was to find which of those two versions of the Imitation Game was the tougher for the machines, it turned out to be 1) above  - the Turing100 at Bletchley Park team reported at a conference [ ICAART 2014 in Angers] - you can view that presentation, 'Fundamental Artificial Intelligence and Machine Performance in Practical Turing tests' here.

Other publications from the 2012 Turing tests include:

Assumptions of Knowledge and the Chinese Room in Turing test Interrogation

Good Machine Performance in Turing's Imitation Game

Human Misidentification in Turing tests

For 2014 the Turing test team from Reading University's School of Systems Engineering and Professor Kevin Warwick (Deputy Vice Chancellor-Research, Coventry University & Visiting Professor Reading University) and Dr. Huma Shah (Research Fellow, previously on RoboLaw and now at the Future Institute, Coventry University) implemented the simultaneous comparison Turing test at The Royal Society, 6-7 June 2014.

A simultaneous-comparison Turing test in which a Judge interrogates a human and a machine in parallel
- image created by Reading University's Chris Chapman

30 Judges, 30 Hidden humans and 5 machines took part across 150 simultaneous comparison Turing tests.

The human participants (judges and hidden humans) were drawn from members of the public and came from all walks of life as well as academia included some who flew in self-funded especially from across Europe, from the US and Russia:

-males and female
-adults and teenagers
- native and non-native English speakers
- a member of the House of Lords
- experts (computer scientists, mathematicians, human language experts), and non-experts (a vet, students, journalists)

The 5 machines involved in 2014 (and also in the 2012 Turing tests at Bletchley Park) were:

Cleverbot - created by Rollo Carpenter

Elbot - created by Fred Roberts

Eugene Goostman - created by Vladimir Veselov and Sergey Ulasen

JFRED - created by Robby Garner

Ultra Hal - created by Robert Medeksza

YouTube of Session 5 Judges in City of London Room 1, and Visitors watching the conversations live on TV screens in the Wellcome Trust Hall in The Royal Society London, 7 June 2014, here:

YouTube of Session 6 Judges in City of London Room 1 in The Royal Society can be seen here:

Pictures from The Colonnade Hotel Maida Vale (was Paddington Lodge when Alan Turing was born there in June 1912):
Add caption

Oh dear my feet got in the way!

Hotel foyer

Turing exhibits in the hotel

Turing plaque at the Colonnade Hotel

Views from my room, above & below in The Colonnade Hotel

One of the best Turing2014 Team memories is staying at Turing's birthplace for one night, before the main/public event, and strolling around Little Venice after a great dinner and discussion with Turing2014 team members.

Pictures from the Royal Society and the Turing tests:

Alan Turing FRS

Reading University, School of System's Engineering RoboLaw Turing test event in the majestic Royal Society London

Judges, hidden-humans and machine developers after Session1

Robotics Professor, Martin Smith, with RoboLaw's Dr. Fiorella Battaglia

Turing test conversation: Session 1

Turing2014 extended team: Lunch on main event day: Saturday 7 June 2014

Hidden humans in Session 5_June 7 2014
Hidden human in Session 5_June 7, 2014
Reading SSE's Nellie Round, and Mr. Geoff Round
Turing2014's Chris Chapman watches over as Judges interrogate

Hidden human in Session 5

Visitors reading the simultaneous Judge+2 hidden entity conversations in the Wellcome Hall

Visitors included Robert Medeksza  (in light blue shirt) of Ultra Hal machine competing in the Turing tests

Visitors crowd around 2 big TV screens displaying simultaneous conversations
Research scientists, Chris Knight and Charlie Moorey at The Royal Society

Judges in City of London Room 1

Alan Turing Year's Professor S.Barry Cooper
Simultaneous comparison Test: Judge's screen

Turing2014 team's Sam Denning and Professor Barry Cooper

City of London Room 1: Judge area

Judge reading information before scoring hidden conversationalists

Dr Federica Lucivero and Professor John Barnden (middle right in hidden-human area)

Dr Fiorella Battaglia next to RoboLaw project documents display

Lunch for the Turing2014 team:

Lunch in City of London Room 2: Saturday June 7

Lunchtime in the Control Room: City of London Room 3: Professor Warwick (left), Mrs Irena Warwick, and Lead Independent Adjudicator, Professor John Barnden of Birmingham University

Professor and Mrs Warwick, Professor John Barnden in experiment control room area: lunchtime

Visitors read the conversations as they happen on two big TV screens
© Albert Efimov

Watching the simultaneous conversations in the Wellcome Hall

Human or machine? Can the Visitors decide!

Reading University School of Systems Engineering student Judging in Session 5

Extreme right: Professor Warwick talking to Fred Roberts of Elbot machine competing in the Turing tests

Professor Aaron Sloman, Turing test Judge at The Royal Society London, 7 June 2014

Journalist Judging in Session 5
From left: Mark Allen (MATT technical support), Professor and Mrs Warwick, + 2 Hidden humans

Hidden human area, Professor Warwick explaining purpose

Visitors and participants relaxing in the Wellcome Hall
Session 6 Judges

Professor Warwick (& Dr. Michael Barclay-visual Turing test, seated)

Score announcement audience in the Wellcome Hall

From left: students from School of Systems Engineering; Eleanor and Emily from Reading University's Events team

2014 Turing test event part of EU RoboLaw 'Emerging Technologies' project

RoboLaw: literature on EU project on emerging technologies behind the event

Turing2014 mug for participants and visitors

RoboLaw- other side of Turing2014 mug

The commemorative bags: forgot to save one for myself!

After the last, 6th Session of the Turing test event, on Saturday 7 June 2014 all the Judge scores were independently checked and verified by Professor John Barnden of Birmingham University, and Dr. Fiorella Battaglia and Dr. Federica Lucivero from the EU RoboLaw project.

Each of the Invited Machine Developers was awarded a one-off RoboLaw/Virtual Robots trophy:
Team behind Eugene Goostman machine: from left:
Igor Bykovskih, Sergey Ulasen, Vladimir Veselov, Andrey Adashchik

Robby Garner_JFRED with his trophy

Fred Roberts_ELBOT with his trophy

Robert Medeksza_Ultra Hal with his trophy

One third of the 30 Judges were convinced by Eugene, scoring it as a human. This does not make the human Judges dumb, it is evidence of the Role of Error-making in Intelligent Thought  - that smart people can be deceived quite easily. Professor Warwick commented:

 "Having a computer that can trick a human into thinking that someone, or even something, is a person we trust is a wake-up call to cybercrime. The Turing Test is a vital tool for combatting that threat. It is important to understand more fully how online, real-time communication of this type can influence an individual human in such a way that they are fooled into believing something is true...when in fact it is not." 

From here.

Finally, comments from some of the people who were there at the historic event -:

Via email:

"Thanks once again for a wonderfully organized event. They get better and better! J
... it was great to be included, especially in such an historic result. I think it will certainly shake up the Turing test debates for quite a while."

 " I would just like to personally thank you for all the effort you clearly put into the event on the 7th and to thank you for allowing me to be involved. It was a fantastic day with a great outcome and I enjoyed every minute. Artificial intelligence is a  fascinating field and a potential world changing technology. I would be happy to be involved or help out in any way possible for future events projects or anything else related to AI."

"Thank you for letting me take part last Saturday. It was great and I really enjoyed it."

"I am so pleased that the event went well."

"Thank you very much for the excellent event and letting me be a part of it. It was thoroughly enjoyable. I’ve been showing off with the items in the goody bag as colleagues had heard about the event on Friday through media. "

"to participate in last weekend's Turing Tests; it was a real privilege to be involved in some way - they've certainly caused a stir."

"take part in the Turing Test. It was very interesting and my grandson ... realy loved it. 
I will be interested to eventually find out the results as he was pretty sure he knew at least one of them was a robot."

Via text message:

"That was amazing"

Twitter comments:

Debunking Eugene: Montreal cognitive scientist Stevan doubts UK university's test claim

. RT : The Test Is Not A Gallup Poll (And It Was Not Passed)

Congratulations on great test experiment 6-7 June 2014 stimulated great discussion!
it was a real pleasure. I was honoured to be involved at all even in such a small way.
Blurring the lines - Eugene is the first program to pass the Turing Test
Celebrity judges and a "child" computer? Is this some kind of joke? Disgraceful.
is this an Ubuntu screen?
What an amazing day and great to meet up with people that I normally only speak to online. Well done to all.
when are you going to release some transcripts? Thanks
Got a link to the transcripts? Or sample chat from the Turing Test winner?

Newspaper and online magazine reports of the Turing test event include:

Slate article by David Auerbach pointing out what the Turing test actually isn't:

"Hunch CEO Chris Dixon tweeted, “The point of the Turing Test is that you pass it when you've built machines that can fully simulate human thinking.” No, that is precisely not how you pass the Turing test. You pass the Turing test by convincing judges that a computer program is human."  

Professor Kevin Warwick on 'How the Turing test was won and why it matters' in The Independent Voice :

Robert Llewellyn, one of the Turing Test Judges on Day 1: 6 June 2014, in the Guardian on his experience at The Royal Society:  http://www.theguardian.com/science/2014/jun/09/turing-test-eugene-goostman

BBC Radio 4 Today: John Humphrys interviews Eugene Goostman 11 June 2014:
http://www.bbc.co.uk/programmes/p020rrgx  and  http://www.bbc.co.uk/programmes/b006qj9z

BBC News: http://www.bbc.co.uk/news/technology-27762088

Gizmodo: http://gizmodo.com/this-is-the-first-computer-in-history-to-have-passed-th-1587780232

NBC News: http://www.nbcnews.com/tech/tech-news/turing-test-computer-program-convinces-judges-its-human-n125786

The Guardian:



The Daily Telegraph:

The Independent:

The Verge:


iFree press release; http://www.i-free.com/en/press/news/6052

BBC London report from Friday 6 June


Report & pictures from Friday 6 June here


Animation for the event: https://www.youtube.com/watch?v=0Hgw9RVwbaw&feature=youtu.be

*For people wondering about methodology of the Royal Society Turing tests, see Warwick & Shah paper in IEEE Transactions on Computational Intelligence and AI in Games: 'Good Machine Performance in Turing's Imitation Game'  and also 'Human Misidentification in Turing tests' available to download for free (as at 15 June 2014) from Journal of Experimental and Theoretical AI here:


For people seeking transcripts of the conversations from the Royal Society tests, please note along with the Judges' scores these will be submitted in peer-reviewed scientific journals and conferences. Please be patient and note Professor Warwick's comment:

"As you might imagine we are yet to unravel the transcripts but when we do these will become available via the normal academic route through academic papers, with our commentary as support. When the papers appear so others will be able to examine the transcripts and see why 33.3% of the interrogators were convinced. We will most likely present each of the transcripts alongside their corresponding hidden human transcript as this is an important part of the tests."

We welcome researchers with different views on what the Turing test is to stage their own public Turing test event. Our interpretation comes from Huma Shah's PhD: Deception-detection and machine intelligence in Practical Turing tests, see also blog post What is Turing test success?

© Dr. Huma Shah 10 June 2014   - content (pictures/text) may be copied with acknowledgement.
[Revised 11 June 2014]
[Updated 14 June 2014, with Slate article link & more photos]
[2nd Update 15 June 2014: more photos and link to 'Human Misidentification in Turing tests' paper in Journal of Theoretical and Experimental AI]
[3rd Update: 27 June 2014: added more photos including of Robert Medeksza of Ultra Hal from Zabaware ]
[4th Update: 29 June 2014: adding Stevan Harnad's Tweets; Like everyone else, he hasn't been involved in a simultaneous comparison Turing test with a human and Eugene Goostman]
[5th Update 6 October 2014: added photograph of team behind development of Eugene Goostman machine]

No comments:

Post a Comment