SignLanguageClusterOfExcellence It can be challenging for deaf people to learn a sound-based language, since they are physically unable to hear those sounds. Hence, many of them struggle with written language as well as with text reading and comprehension. Therefore, a good deal of website content presents challenges for them. Computer scientists from Saarbrücken, Germany, want to change the situation by means of a method they developed: animated online characters display content in sign language. In the long term, deaf people would be able to utilize avatars to communicate on online platforms via sign language. To realize the technique, users would only need readily available devices.

Sign language was first acknowledged as a separate language only in the 1960s. Similar to spoken language, it evolved from different cultural backgrounds. Every country has its own sign language with various dialects, which are based on different rules than the spoken language.

Although several websites provide video clips in which sign language interpreters translate the text, much Internet content remains cryptic for the deaf community. To inform deaf people quickly in cases where there is no interpreter on hand, researchers are working on a novel approach to provide content. Their idea: avatars. These animated characters could be used in the context of announcements at train stations, or on websites.

“We have already gained initial experiences with avatars,” says Alexis Heloir, who is the leader of the research group “Sign Language Synthesis and Interaction” at the Cluster of Excellence “Multimodal Computing and Interaction” and also a researcher at the German Center for Artificial Intelligence. “If we try to animate them like human beings, deaf people have issues with understanding the avatars.”

This may be due to the greater variety of emotional expressions of humans compared to avatars. To deal with that problem, Heloir and Fabrizio Nunnari are creating avatars that make more accentuated movements. The researchers are closely cooperating with Peter Schaar, who is deaf and is a lecturer for German sign language at the Saarland University Language Center and the College of Engineering and Commerce in Saarbrücken.

“Our method should be inexpensive and easy to use so that every member of the deaf community will be able to use it”, says Nunnari.

To capture the motions of deaf people, the scientists make use of affordable cameras and sensors that are typically used by teenagers for computer games. A computing method transfers the movements of the entire body onto the avatar. In the long term, the researchers want to create a collection of short sign language sequences that can be used by the deaf to interact on the web.

Source: Cluster of Excellence on “Multimodal Computing and Interaction”