Glove converts sign language into sound

Share with others:

Print Email Read Later

Related Media:

It knows only 32 words, but someday, it may get a grip on the entire human vocabulary.

It is a sensor-equipped glove, known as HandTalk , that can translate gestures into spoken words on a cell phone. It was developed by students at Carnegie Mellon University as part of a class research project.

Three of the four team members, senior computer engineering students Bhargav Bhat, Hemant Sikaria and Jorge L. Meza , demonstrated the prototype yesterday at Carnegie Mellon's "Meeting of the Minds" expo of undergraduate research projects.

Someday, the young inventors hope, it may allow deaf people to communicate with those who don't know American Sign Language by having their cell phones speak their words aloud.

"That could be a big advantage" for hearing-impaired people, Mr. Bhat said. "It would cut out the need for an interpreter."

For now, the glove uses a primitive language system invented by the student team, which also includes master's student Wesley Jin .

When the glove is held in a fist, for instance, the cell phone says "Good morning." When the index finger, second finger and thumb are extended, it says, "I'm having a good time." And when the index finger, little finger and thumb are held out, it politely says, "Thank you for your time."

Underneath the hood of this system are several relatively inexpensive pieces of technology.

Along each finger and the thumb of the glove are flexor strips, which change their electrical resistance, depending on how much the digits are curled. The positions of the fingers are read by a chip and transmitted wirelessly to a cell phone, which is loaded with a vocabulary that corresponds to the gestures.

The cell phone then types the words as text messages, and an off-the-shelf program translates them into speech.

Mr. Bhat said HandTalk so far has been able to learn 15 of the 26 letters in the American Sign Language alphabet.

To learn the others, though, the team will have to add pressure sensors and accelerometers to the glove to determine when fingers are touching and how much the hand is rotating.

And to fully accommodate ASL, the system will have to use two gloves and measure the relative position of both hands.

On top of that, the team needs to learn how to adjust the gloves to each user, said electrical and computer engineering professor Priya Narasimhan , the team's adviser. One person may make a gesture with their hand at a 45-degree angle, while another may do the same at 60 degrees, she said, " so their biggest technical hurdle in developing this is the calibration."

The team hopes to begin testing the gloves with hearing-impaired people in about three or four months, Mr. Sikaria said.

The HandTalk project is one of several created this year in Dr. Narasimhan's Embedded System Design course, in which teams of four have to develop a product prototype in 15 weeks.

Among the other projects this term is a lighted jump rope that changes colors the faster someone skips and can deliver digital messages on how many calories are being burned, and a cell phone system that allows students sitting "in the nosebleed section" of a hockey game to tap into any video camera feed within the building and see it on their phone screens.

One project from last year's class already has resulted in a local spin-off company, Dr. Narasimhan said. The students developed a bar code reader for blind people that gets product and price information from the Web and reads it back to the shopper.

That project helped spawn HandTalk, she said.

"We started to look at assistive technologies for a wider range of people with disabilities, and one thing was obviously people who were deaf, who communicate with sign language, and it's interesting to be able to support them talking to people who don't know sign language through technology."

The technology itself would lend the gloves to several other uses, the students said. They could be used as game controllers, remote controls and even as a way for doctors to check how much hand function a patient had regained after an injury.

The students even rigged up HandTalk to play a tune by moving the fingers. And the instrument?

The cell phone keypad, of course.

Mark Roth can be reached at or at 412-263-1130 First Published May 8, 2008 4:00 AM


Create a free PG account.
Already have an account?