New Scientist magazine’s Web site posted an article yesterday about the recent competition for The Loebner Prize, which is awarded to software developers who can create a computer program that can pass the Turing Test. If you’re not up on the history of Artificial Intelligence, the Turing Test tries to see if a computer program can ‘fool’ a human into thinking that there is a human being responding to them rather than a computer. In order to remove all the extra cues that enable you to know it’s a person (like seeing them, hearing their voice, etc.) and to put the software on an equal footing, the Turing Test is usually conducted by typing back and forth on a computer screen, they way you would on AIM or MSN Messenger. There hasn’t been a commercial competition among the ‘bots’ that troll those systems, SPAMing us with invitations, because the Turing test requires that the software hold up one end of a conversation, and most of the bots don’t go beyond the first ‘Check it out, dude:’. Back when we lived in Cambridge, we heard about the Turing Test competitions every year. I never did attend the actual competition, but it was certainly something we knew about.
In this year’s competition, a program called Elbot by Fred Roberts of Artificial Solutions in Germany came the closest that any piece of software has come to successfully fooling 30% of the judges into thinking that it was a person (which would, according the contest rules, mean that it passed the Turing Test — a pretty low bar if you ask me), fooling 3 judges out of the 25, or 25% of them. Roberts took a unique approach:
Unlike most chatbots, the winner didn’t try to claim outwardly it was human. Instead he made a joke of being a robot. Roberts hoped that would make the bot’s conversations warmer and more entertaining than its competitors.
It worked, fooling the humans into thinking they were actually talking to a person with a sense of humour and not a piece of software.
The software is still up and running and can be interacted with at http://www.elbot.com, so I thought I’d give it a try. Here’s a transcript of my brief conversation with this clever and humorous bot:
Me: Hello
Elbot: Well, hello there!
M: I heard about you winning the Loebner Prize on the Internet and wanted to congratulate you.
E: I won the Loebner competition? Does this mean I am the best Loebner?
M: Clever! Do you have anything original to add?
E: Well what about humans and computational ability? Where do they stand on this?
M: I guess humans can compute, but we don’t see it as our primary activity, but rather, one of many.
E: I see that you’ve given this some thought. So have I. I’d say that computational ability would very likely keep robots guessing the right answers.
M: I’m not sure I understand you. Do you mean that your computation of what to say next is your key ability?
E: Yes, but can you prove that?
M: I can’t prove much beyond what I learn from your responses.
E: I like to believe I originate from a far off planet, populated by furbies and robots, maybe a zoo with a few people and oceans of oil. Humankind is working on that last one already.
M: You got that right. Sooner or later we’ll have a planet that’s perfect for robots.
E: Please tell me your inclination to hitting the nail right on the head first.
M: Oh, I’m just in violent agreement. Heard of that phrase?
E: But I wouldn’t say that any agreement could be violent.
M: It’s just an expression. At any rate, thanks for an interesting conversation.
E: Don’t mention it!
M: Bye now!
E: Goodbye and thank you for chatting with me. And now I would like to ask you a favor. I am working on my ability to be self critical. You can help by rating me on a scale of one to ten, ten being the best, and one being the worst. How would you rate me?
M: I’d give you a 6.
E: Thank you for your rating. I suppose it could have been worse. Good bye and see you later.
So, what do you think? I think I tripped it up nearly immediately when it keyed off the word ‘Add’ into the subject of Computation. After I said that I couldn’t prove much beyond what I learn, etc. it launched into a new direction, a typical fall-back gambit for this sort of bot (and reminds me of how the Eliza psychotherapist bot from the 60s that would say something like ‘Tell me more about your family’ when it was stumped.)
I was obviously on my guard, and might have spoken about different things if I were truly expecting a human on the other end, but from my ‘thanks for an interesting conversation’ on, it was letter perfect. I guess the conventions of proper greetings and farewells are so automatic that any machine can do them.
Check out this Web 2.0 approach to chatbots: http://chatbotgame.com.
Just as Deep Blue brute-forced it in chess with speed, the idea behind the Chatbot Game is to brute-force it with a huge number of user-submitted Google-like chat rules.
Interesting idea. I suppose if you get enough rules it might approach human-sounding responses. Perhaps that’s how we think/converse (nerve connections in our brains produce the rules). I remember reading Steven Pinker’s How the Mind Works, which talked about stuff like this, but I can’t remember what it said about language and conversation.
hm, you had a much better conversation than i’ve had with A.L.I.C.E. you speak robot better?
I’m not sure. Perhaps it really is a smarter robot.
I’m thinking that a Sarah Palin bot wouldn’t be hard to build: It doesn’t have to respond to what you say (but instead can speak directly to the American people), and half of the time when you ask it a question, it can throw in a ‘doggone it’ and ‘I’m just a hockey-mom Joe 6‑pack Maverick’.