Microsoft Research Uses Kinect To Translate Between Spoken and Sign Languages 79
An anonymous reader writes in with a neat project Microsoft is working on to translate sign language with a Kinect. "Microsoft Research is now using the Kinect to bridge the gap between folks who don't speak the same language, whether they can hear or not. The Kinect Sign Language Translator is a research prototype that can translate sign language into spoken language and vice versa. The best part? It does it all in real time."
P2TC signs: "I really like this" (Score:3)
Kinect translation with new autocorrect-for-ASL: "I really like your tits"
Thanks Microsoft!
Re: (Score:2)
As a /.'er unfamiliar with ASL, could you elaborate? I learned a couple of phrases for some volunteer work I did a few years back, but nothing to do with tits. I feel like I missed the most important part of the lesson.
Re: (Score:2)
Re: (Score:1)
This little joke actually raises an interesting topic, as sign language usually does not follow the same grammar as spoken/written language. Sign languages can differ from country to country or from region to region, similar to dialects. The hands aren't the only things used to convey meanings either: facial expressions or body movements are used too.
An example of sign language using the line above could be to represent the words "breasts", "you" and "like", in that order, with expressing the meaning "very
P2TC signs: "I really like this" (Score:2)
nods and looks sad, grabs cap brim with left hand and runs right thumb along chin
And vice versa? (Score:2)
How do you sign: Dear aunt, let's set so double the killer delete select all?
It probably wouldn't, because... (Score:2)
some signs need no translation.
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
In ASL the bird is directional. Not sure about CSL. But you don't point it upward in ASL, you point it as the moron you're insulting.
Re: (Score:2)
I'd love to see a blind person try to use touch screen phone.
Re: (Score:3, Interesting)
never underestimate what people can do
http://appleinsider.com/articles/11/09/15/stevie_wonder_thanks_steve_jobs_for_making_ios_devices_fully_accessible [appleinsider.com]
Re:Sign Language Is Obsolete (Score:4, Insightful)
Re: (Score:1)
Have... Have you heard of Helen Keller?
Re:Sign Language Is Obsolete (Score:5, Funny)
Re: (Score:2)
You'd see them hold their hands around the person signing to them, so they can feel the gestures. This is one of several techniques [aadb.org].
Re: (Score:2)
They do. My friend did a article about that. She spend one week with a couple who both are deaf and the man had become blind few years before.
They both communicate with sign language but it is done by holding each other hands when doing it. There were three couples registered to be in same situation that they need to communicate with physical sign language.
You can be amazed what people can do who can not hear, talk and don't have anymore capability to see. They live normal live just together.
ps. My friend i
Re:Sign Language Is Obsolete (Score:5, Informative)
They do. They use iPhones (and iPads). iOS has an accessibility feature called VoiceOver which changes the input paradigm from a touch initiating a 'click' action to one where a touch reads out the description of the UI element with text-to-speech. Two taps on the item will send a tap to the UI, and a three-finger swipe will initiate scrolling. So you can basically drag your finger across the screen and find things with your ears instead of your eyes, then navigate and interact as a sighted person would once you get your bearings.
Re: Sign Language Is Obsolete (Score:1)
Can't tell if that's intended to be a troll or not. Since this is the internet I have to assume it is.
As far as trolling goes, being married to a deaf women I'm pretty pissed off at it. It's so rude and dead wrong I'm left incredulous, but that is the point of trolling isn't it.
Re: (Score:3)
I'd love to see a blind person try to use touch screen phone.
Touch screen phones may well be the best tech to come along for helping blind people ever.
http://bits.blogs.nytimes.com/2013/09/29/disruptions-guided-by-touch-screens-blind-turn-to-smartphones-for-sight [nytimes.com]
Since that might be pay-walled, here's a copy:
Disruptions: Visually Impaired Turn to Smartphones to See Their World
September 29, 2013, 11:00 am
In recent years, many smartphone apps that are aimed at blind people have appeared.
Luis Perez loves taking photographs. He shoots mostly on an iPhone, snapping gorgeou
Re: (Score:2)
Not just text messages, but also online college courses, Facebook messages, etc. Especially for people like me who are deaf enough that suffer a lot - socially, academically and in the workplace, but not so deaf that we qualify for any sort of assistance. It's a shame Obamacare doesn't address hearing aids and glasses, it would have likely gotten a lot more support.
Re: (Score:3, Interesting)
It's a shame Obamacare doesn't address hearing aids and glasses, it would have likely gotten a lot more support.
That's funny. Not only is the deaf community a tiny minority (unlikely to have any impact at all), the issue of hearing aids is enough to divide them! If they found out you could get a cochlear implant with insurance purchased through an exchange, you'd see little other than opposition. (Yeah, they're that crazy.)
Not that they're likely to be aware of the issue, as illiteracy is so prevalent. Still, if they found out, they'd oppose it.
Re: (Score:2, Insightful)
Re: (Score:2)
That's generally a vocal minority that's mostly perpetuated by the older population that has always been deaf. Deaf people who aren't exposed to the older deaf generation are much more open minded. Unfortunately, some colleges aimed at the deaf (I'm looking at you, RIT NTID) continue to perpetuate this - as a result, many students there come to equate deafness with being an asshole. And I don't blame them.
Re: (Score:2)
People who are functionally deaf are a minority (less than a million in the US), and hearing aids will likely never help them. People who are partially deaf and would benefit from hearing aids are thought to number at least 10 million in the US, and that may be a low estimate. To put it in comparison, there are only 7 states with more than 10 million residents (and three of those aren't too much higher).
There's also a lot of people with partial hearing loss who do not identify with any "deaf community". I r
Re:Sign Language Is Obsolete (Score:4, Insightful)
You clearly haven't seen two or more deaf people in public.
Plenty of people sign. Heck, plenty of hearing people sign.
Conversations (deaf-deaf, hearing-hearing, hearing-deaf) are all much more natural face-to-face where you can interrupt, show expressions, assign nuanced gestures or tones to words -- all things that are not well suited for text.
Re: (Score:2)
Spot on. Also, sign language is quite important culturally as the native language of deaf people. All cultures view their native language as essential to their cultural identity. The deaf are no different (or so I hear.)
Re: (Score:2)
Except sign is a con lang -- strike that -- a bunch of different con langs incompatible with one another.
Re: (Score:2)
Sign comes in handy in very noisy environments, or conversely where silence is golden. Then there's job-specific signing, as for crane operators and their spotters.
Comment removed (Score:5, Informative)
Re: (Score:2)
Deaf culture is destructive.
Re: (Score:1)
Well... sending each other texts while standing next to each other works for non-deaf teens....
Re: (Score:1)
This is one of the stupidest things I've read all day.
Awfully generous of you to use "day."
Re: (Score:2, Informative)
Your comment shows the sign of ignorance. Plenty of deaf AND hearing people sign anywhere. It's easily 10 times faster than texting. Even faster than speech itself.
Re: (Score:2)
Yeah, I've found that presenting the proper (or improper) finger while driving is much faster than yelling "F- off" out the window.
Re: (Score:2)
Re:Sign Language Is Obsolete (Score:4, Informative)
Re: (Score:3)
it is quicker and more convenient for the disabled to send a text message.
I would have thought it would be more convenient for someone to speak/be spoken to in their native language - sign - rather than send/receive a text in a foreign language - English or whatever.
ASL? Siglish? (Score:2)
There's a lot of contextual clues necessary to understand sign language. Most conversations would seem "faux caveman" like to the outsider - a lot of Noun Verb Noun going on...
I'm going to have to watch the video from another machine, but I'm more interested in the bumper at the bottom that has realtime English/Chinese translation in your own voice...
Re: (Score:2)
They (I assume the same group) demoed the real-time English/Chinese translation in your own voice last year. It's really impressive, and the results were surprisingly good.
I do wonder how it deals with phonemes that are present in one language but not in another, maybe there's a "training process" you have to do initially to make sure it has enough recorded samples to get full-coverage of the target language.
Hardly revolutionary (Score:4, Interesting)
We were doing real-time ASL translation to text using the webcam on the Indy2 workstation back in 1997, success rate was about 85% and most of the misses were from hidden object problems which the Kinect does nothing to help with.
Re: (Score:3)
Believe it or not, the video compression codec we now know as h.264 had its origins about 20 years ago as a compression scheme to allow the transmission of video good enough to allow sign language over a dialup modem. Its evolution into h.264 (along with some very powerful ASICs that jumpstarted h.264 to allow realtime hardware compression and decompression whose development costs were subsidized by the federal government) perfectly demonstrates an assertion I've long made -- the best way for deaf people to
Re: (Score:2)
Hmm. I know Morton Nadler at Virgina Tech was working on a system to image-process, recognize, compress and transmit sign language (I think it may have been fingerspelling) back in the early- to mid-1980s. I think he was using racks of custom hardware; this may have pre-dated even FPGAs. It was expensive, but everybody knew that hardware would keep getting cheaper...
Re: (Score:2)
Re: (Score:2)
And these people are NOT doing real time translation. They're having people sign each sign stilted isolation.
As
Though
They
Talked
Like
This
Edutainment (Score:1)
Re: (Score:2)
It would probably be invaluable for video-chatting/conferences (or even IRL discussions) where there's a mix of Deaf people &hearing ones that don't know the specific sign language they use, as it'd mean the two groups could interact directly/independently without requiring a translator. That'd be great for private sensitive conversations between two individuals, relatives meeting for the first time (as it'd mean they could communicate directly), and might have a good impact on Deaf people's employabi
There's an interesting problem with this approach (Score:2)
Needless to say, I though it was too hard to solve the problem on a smartphone so I postponed the project. I don't think Kinect can do a much better job at picking up the
Re: (Score:2)
UNlikely.
Much subtler changes for facial work.
Re: (Score:2)
Yup. Many signed phrases are just noun, verb, point, noun .
Re: (Score:1)
They didn't immediately make it able to understand full-speed sign language including that from a man wearing black gloves in a dark room?!!!eleven!
Curse you, Microsoft Research, curse you to the depths of fucking hell!!!!
Very creative (Score:1)
There are lots of research being done with kinect, by BS and masters students, mostly around physiotherapy. This is one of those creative applications that everybody says after hearing about it .. "damn, why didn't I think of that". Very creative use of the kinect.
Real Time? (Score:2)
Looking at the video in the article, it seems that "in real time" means "at about 1/4 of the speed of regular signing".
Imagine. Having. To. Speak. Like. Kirk.
Re: (Score:2)
OK, you have a point.
Being understood definitely beats not being, even if you have to slow down to achieve that. And the translation was happening reasonably quickly if you were slow enough.
I was just a bit disappointed since from the "real time" I was expecting the translation to keep up with someone speaking at normal conversational speed. (Like the asker at the information kiosk, for example.)
Most impressive use of technology I've ever seen (Score:5, Interesting)
Disclaimer: yes I work for Microsoft. No not on these projects.
This was demo'd live in front of 30K MSFT employees at our annual company meeting. It nearly brought me to tears. Yes, I can see through demoware and and yes it's highly imperfect, but honestly it was the single most impressive use of technology I've ever seen. It was both novel and simple. It combined hardware, algorithms, user experience, and cloud scale. I don't know if it will ever go anywhere though I expect that it will. The key point here is that these are off the shelf components. Kinect and gesture APIs combined with machine translation and text to speech. It's important that these are, all or nearly all public production APIs. Such a system 10 years ago even if possible, would never make it to market because of the tiny user base. Today we can build such apps for the 0.01% of the population that need Mandarin Sign Language translated to English. And it can be cost effectively. That is the point. Technology being used to address real problems for under served communities. So yes, maybe people researched automated sign language recognition years ago, but bringing it to market and enabling a scenario for real people is a wholly different beast
Universal Translator (Score:2)
I knew it! the Federation runs on Microsoft technology. Guess that explains all the exploding consoles and Transporter accidents.
Way too many Dialects (Score:2)
The ASL interpreters I know do a lot of on-call work for medical, mental health and educational purposes. One thing they mentioned is ASL has regional dialects.