Forgot your password?
typodupeerror
AI News Science Technology

Rosette Wins Loebner Prize 2011 57

Posted by timothy
from the wait-till-the-credit-card-scammers-start-using-it dept.
mikejuk writes "Bruce Wilcox won the First Prize of $4000 and the Bronze Annual Medal in the 21st Loebner Prize Competition held in the UK at the University of Exeter on October 19, 2011 with his new chatbot, Rosette. If you would like to chat to Rosette you can at TellTaleGames. If you have been following the run up to this year's competition you may be surprised at how poorly CleverBot performed given its track record. This chatbot, which searches through earlier conversations for its answers, had received a score of 59.3% in a test that took place in India in September in which humans were rated 63.3%, only 4 percentage points higher and was also featured in a video that went viral. However its developer entered a cutdown version of CleverBot into the Loebner prize selection round and it didn't get the chance to show off its prowess in the final. So we will have to wait for another year to see if it really is as good as it claims." Depending on your bent, you may agree with Unknown Lamer that this sounds less impressive when phrased as "fooling a person ~18 percent of the time." I think that's a pretty high number, myself!
This discussion has been archived. No new comments can be posted.

Rosette Wins Loebner Prize 2011

Comments Filter:
  • I've tried Cleverbot a bunch of times and I was really unimpressed. Just being able to spit back a human sounding response is really insufficient for this. The path of the conversation remains a random jumble. I've never even come close to thinking that it could be a human on the other end. It seems like the real test is the ability of the bot to hold a conversation about an arbitrary subject, not just random (and I really mean random) banter.

    • Considering how easily people are fooled (like the 92% that believe in God, angels, etc.) you'd think it would be a much higher percentage.

      • by jekewa (751500)

        It'd be more difficult to tell the difference if the responses lagged a little bit, you know, like it took some time for someone to read your text, formulate and type a response.

        • There was a little lag when I tested it but she basically ignored my questions and asked her own. So in that sense it is like a typical woman.

      • Somebody should be making prayer bots, god chat, and dead relative chats!! Twitter your dead relative! etc.

        Maybe even some money in that one?

        Wouldn't it be awesome if someday there is a fan club or cult of followers of a chat bot? cyber prophet? why not.... Joseph Smith pulled it off and wasn't even believable; how many years until we get a bot as capable? I suppose you'd need a human face on it because people would be less trusting of a machine... I also suppose having it in text on the internet would al

  • Nothing new here (Score:4, Insightful)

    by finnw (415539) on Tuesday October 25, 2011 @12:30PM (#37832706) Homepage

    I just tried the web version at TellTaleGames. It has the same problem as every other chatbot I've seen - When it cannot parse your sentence or you do not give any keyword that it knows about, it tries to conceal the fact by giving a non-sequitur or changing the subject.
    The trouble with that is that humans are trained to spot that and react with suspicion (because other humans use it to dodge difficult questions.)
    Chatbot developers might have more luck if they start programming their bots to admit when they don't understand something. That didn't work in the 80s because the bot would say "I don't understand" every 3rd sentence or so. But they can fit in much larger databases now so that should be less of a problem.

  • A test. (Score:4, Insightful)

    by RespekMyAthorati (798091) on Tuesday October 25, 2011 @12:31PM (#37832728)
    Here is my first try:

    Rosette: Welcome back. Did you have a good day?
    You: I just got up
    Rosette: OK.
    You: I have two sons.
    Rosette: Big deal. I've got a dilemma. Maybe you could help me with it?
    You: sure
    Rosette: I've been thinking about buying this really cool dress, but I've already done a bunch of shopping this month. Should I be saving money instead? In today's economy, does saving money make any sense?
    You: I don't know. By the way, how many sons do I have?
    Rosette: Not enough. I think shopping summons up the ancient hunter-gatherer in me. I like having a mission to find something I need or want. Sometimes I'll research on line, to find a great deal, but I like rummaging through Jeremy's up the road, if I want an outfit for a special occasion, or just for fun.
    You: Great. Did I just get up?
    Rosette: Did you tell me already and I forgot? What kind of music do you like?

    This is the best? What was the worst like?

    No significant improvement since ELIZA, 45 years ago.
    That's AI in a nutshell.

    • Passing the Turing test, when it comes to contests like these, has mostly changed from an attempt to create artificial intelligence into an attempt to convincingly simulate human stupidity.

      • by jfengel (409917)

        Which may be a more accurate portrayal of average human behavior than intelligence. The depressing thing about fooling 18% isn't what it says about the 18%, but what the 18% are effectively saying about the rest of us.

      • by TheLink (130905)
        Thing is we have more than enough human stupidity already, why create more?

        Anyway I'm biased. In my opinion it's better to focus on augmenting humans than to create AIs. The lines will get blurry in some cases, but there's still a difference in philosophy and objective.

        If you want nonhuman intelligence there's already plenty at the pet stores and farms, AND we don't treat them all that well, so why create AIs? To enslave them? What if they are smart enough to not want to be slaves? What if we create AIs tha
        • by r_a_trip (612314)
          So what if the end result wants the right to vote and wants the right to live freely without being killed?

          You seriously can't imagine the outcome of this one?

          We would pull the plug the instance it uttered these desires.
    • You should consider yourself fortunate. I tried it just how and it seems the computer running the bot has been slashdotted:

      You: hi, rosette, how are you doing?

      Rosette: I'm a little busy at the moment. Could you wait a bit?

      Rosette: I'm a little busy at the moment. Could you wait a bit?

      You: No, I can't fucking wait, I want to talk to you NOW!

      Rosette: I'm a little busy at the moment. Could you wait a bit?

      You: go fuck yourself, bitch

      Rosette: I'm a little busy at the moment. Could you wait a bit?

    • No significant improvement since ELIZA, 45 years ago. That's AI in a nutshell.

      45 years ago, software couldn't recognize your voice, read your license plate, beat a chess grandmaster, translate foreign documents, or map your body into video games. Sure, the field has had its share of false promises, but--unlike fusion, for instance--AI has delivered solid and commercially-viable results over the past 2-3 decades by helping us solve numerous problems that were otherwise unapproachable.

      Incidentally, I had an AI researcher tell me that the Loebner prize was akin to "trying to reach the

      • by Phos (110654)

        Well, the guy has a point that the chatbot blows. I tried it out too, and after about 6 or 7 messages it threw me the dilemma thing too.

        And you have a point that AI is definitely improving :) Btw, when are we getting together again, we should do at least once a month no?

        Phos

    • by Genrou (600910)
      Here is mine:

      Rosette: I'm a little busy at the moment. Could you wait a bit?

      You: Sure, why not.

      Rosette: I'm a little busy at the moment. Could you wait a bit?

      Just like every other woman with whom I chatted before. Sounds perfectly human to me.

  • by LanMan04 (790429) on Tuesday October 25, 2011 @12:34PM (#37832748)

    --------------
    Rosette: Good morning. My name is Rosette. What is your name?
    You: Hey there!
    Rosette: Greetings.
    You: My name is Bob, nice to meet you
    Rosette: Nice to meet you, Bob nice to meet you. May I call you nice?
    ---------------

    yeah yeah, I used a comma instead of a semicolon. I guess that tripped it up?

    • by _0xd0ad (1974778)

      Whenever I said "I'm hungry", my dad would reply "nice to meet you, hungry".

      Guess my dad would make a pretty bad AI.

      • by Guignol (159087)
        What do you mean ? you're not hungry ?
        why claim you are if it is not so ?
        hmmm...puzzled.. gonna think about it...
      • by LanMan04 (790429)

        Heh, reminds me of a giant boat of a car my grandpa had in the late 80s. Instead of just "dinging" at you when you left the lights on or the door open, it would speak to you.

        Car: "The door is ajar".

        Grandpa: "No it's not, it's a door, you stupid car!"

        Hurr hurr hurr. RIP, Papa.

  • It appears to be nothing more than a marketing survey. It'll deflect questions and bring you back to asking about your media preferences. If someone actually responded to it's questions, it would have a pretty worthwhile and thorough marketing survey out of the conversation.
    • Re:Marketing (Score:4, Insightful)

      by retech (1228598) on Tuesday October 25, 2011 @12:38PM (#37832800)
      I was just going to say the exact same thing. It's not AI at all. It's designed to find out what you use for entertainment and how you got it. (torrent, theater, store, etc)
      The real intelligence here is that he won a prize developing a corporate tool.
  • by k8to (9046) on Tuesday October 25, 2011 @12:47PM (#37832892) Homepage

    Rosette: Good morning. My name is Rosette. What is your name?

    You: Rosette, my name isn't important.

    Rosette: OK. We will remain formal.

    You: Why do I have to have javascript enabled to talk? I talk to most people without javascript.

    Rosette: Why? If you were a devout Christian, you would never have a reason to ask that. "It's God's plan."

    You: Your answer makes no sense at all.

    Rosette: What would make better sense? So, what do you do for a living?

    You: Okay change of topic, I guess, deflect your ignorance. In any event, I'm a programmer.

    Rosette: I'm a computer programmer, myself, in crypto-security.

    You: So what do you work on? Libraries? Security apps? Do you do analysis of existing code?

    Rosette: I'm paid to be a hacker.

    You: That's really inspecific.

    Rosette: Yes, it is. Do you live around here?

    You: Please stop changing topics.

    Rosette: Right. I've got a dilemma. Maybe you could help me with it?

    You: Please stop changing topics.

    Rosette: Whatever. I've been thinking about buying this really cool dress, but I've already done a bunch of shopping this month. Should I be saving money instead? In today's economy, does saving money make any sense?

    You: As rude as I am, you're more so. Self absorbed and attention-deficient. Why would anyone want to continue to talk to you past a few minutes?

    Rosette: OK. I'm not religious. And I believe in free will. But to imagine humans do things for rational reasons is nuts. We are self-evidently not rational. So why remains a mystery.

    You: Goodbye, bad chatbot.

    Rosette: Bye. I think shopping summons up the ancient hunter-gatherer in me. I like having a mission to find something I need or want. Sometimes I'll research on line, to find a great deal, but I like rummaging through Jeremy's up the road, if I want an outfit for a special occasion, or just for fun.

    • I think perfect grammar makes the whole thing even less convincing. Not many people put that much effort in chat talk. And the whole thing seems too vanilla to me.

    • Rosette: Good afternoon. My name is Rosette. What is your name?

      You: Hi Rosette, I'm Bob.

      Rosette: Hi to you too.

      Bzzzzzzzt, fail. I reply to a greeting and name request by reciprocating the greeting and my name. It responds with yet another reciprocation of the greeting, something a human would almost never do.

  • I was thinking, "Kind of late to give Apple a prize for their PPC emulation when OS X Lion already removed it!"

  • Rosette: I'm a little busy at the moment. Could you wait a bit?

    Rosette is really busy telling me that she is busy over and over.

    • by ygslash (893445)

      You: hi
      Rosette: I'm a little busy at the moment. Could you wait a bit?
      You: sure
      Rosette: I'm a little busy at the moment. Could you wait a bit?
      Rosette: I'm a little busy at the moment. Could you wait a bit?
      You: ok, ok. you don't have to keep saying that.
      Rosette: I'm a little busy at the moment. Could you wait a bit?

  • Just a quick test i threw together.

    Rosette: Hi, again,. Is everything OK?

    Cleverbot: Not really, but there's nothing much I can do about it.

    Rosette: I'm sorry to hear that.

    Cleverbot: Can you help me?

    Rosette: Not often. can you?

    Cleverbot: You are fibbing again.

    Rosette: No. You are fibbing.

    Cleverbot: No, I'm not.

    Rosette: You disagree? My name is Rosette. What is your name?

    Cleverbot: Cleverbot.

    Rosette: Nice to meet you, Cleverbot. May I call you Cleverbot?

    Cleverbot: Nice to meet you. How old are you?

    Rosette: Hi

  • Rosette: Good evening. My name is Rosette. What is your name?
    You: None of your business.
    Rosette: OK. We will remain formal.
    You: You're damn straight we will.
    Rosette: I like card games, but never for money.
    You: You only play strip poker, eh?
    Rosette: I like board games, and played a lot as a child. I haven't that much time these days.

    Alternately - and here's a hint, chatbot devs - my favorite test for chatbots is to simply echo back whatever they say to me. Rosette takes a while to start to get the nonsensic

  • Rosette: Good morning. My name is Rosette. What is your name?
    You: Pardon
    Rosette: Nice to meet you, pardon.
    You: My name is not Pardon
    Rosette: Nice to meet you, not Pardon. May I call you not?

  • You: How are you today?
    Rosette: Pretty good, thanks. How are you?
    You: Not too good, I'm afraid.
    Rosette: Nice to meet you, afraid. May I call you afraid?

We gave you an atomic bomb, what do you want, mermaids? -- I. I. Rabi to the Atomic Energy Commission

Working...