SBIR-STTR-Success: Speechgear

TALKING SENSE: Translation technologies support civilian travelers, warfighters, and the hearing impaired

Now that we all walk around with miniature computers in our pockets, it can be harder to get excited about amazing new technology. Yet, once in a while, a company will come along and deliver an innovation that gives us a tingle up the spine.

Minnesota-based SpeechGear has come up with one of those innovations, and it all started with a Small Business Innovation Research (SBIR) contract beginning in 2001. Just before 9/11—in April, to be exact— Robert Palmquist proposed a speech translation system to the Marine Corps and the Office of Naval Research. Almost presciently, he proposed the first translation system be between Arabic and English.

But don’t the Marines just use native interpreters? As it happens, native interpreters can be problematic. A Marine might say, “Do you know where the bad guy is?” And instead of translating the question, the translator could say, “Just shake your head and say ‘No.’” Native translators can also potentially feed information to the enemy. Allowing a warfighter to talk directly to civilians has obvious benefits. Fortunately, Palmquist and SpeechGear were able to have a working translation prototype ready in September of 2001. It was used successfully in Iraq.

Since then, many companies have come out with (and even more have promised) translation capabilities, but Robert Palmquist, who has a Master’s degree in Mechanical Engineering and Computer Science, is credited as the first individual to develop and deploy a working translation system. His approach to creating such a system is unique in that it uses neural net technology. He was the first to get utility patents for this approach, and thanks to the SBIR funding, the US government has rights to some of those patents.

In the years after the first Gulf War, SpeechGear continued to receive SBIR funding, and the translation product has evolved to meet the needs of warriors as they come home with hearing loss. Wearing hearing protection means losing situational awareness, so Marines often choose not to wear it. To help support them, Speech Gear’s Interact-AS™ provides real-time captioning, translating English to English.

Just as the software now helps the hearing impaired, SpeechGear has adapted to serve the civilian marketplace. A separate company, Auditory Sciences, was spun off to focus on communication products for the hard-of-hearing and deaf. As it turns out, captioning is needed in a lot of places. Adults in the workplace need it because hearing aids don’t work well in noisy environments. People who have difficulty typing, perhaps because of a stroke, can use it to generate text for documents and emails. Schools also have a need, as the Americans with Disabilities Act (ADA) requires them to provide students with hearing loss the same education as their peers.

Karen Anderson, founder of Success for Children with Hearing Loss, has partnered with Auditory Sciences to provide schools with captioning tools. Before these tools were avail-able, schools would hire a court reporter to type everything – a solution that was unaffordable for most. As hearing-impaired students reach secondary school, it becomes harder for them to keep pace, as they have to learn new vocabulary even as they try to understand what is being said. Real-time captioning helps them keep up. The company doesn’t currently take advantage of the translation aspect of the tools, but Karen imagines an English as a Second Language classroom where the teacher speaks English while the students see captions in as many different languages as needed.

Seeing Interact-Streamer in action is impressive. First, it’s fast. You don’t need to download an application to use it; the system is in the cloud, so you only have to go to the Streamer website (though it can be run locally, if necessary). The second thing you’ll notice is that it’s capable of modifying the translation on the fly. In this example, it used the context of the conversation to figure out that we were talking about a microphone rather than someone named Mike:

I have a separate Mike for... I have a separate mic for this captioning system.

Watching it go back and fix mistakes as it better understands the context is fun to watch. Another thing that’s unique – it actually puts in punctuation. SpeechGear loaded all of Wikipedia and 150 English novels into a neural net to create a punctuation engine. It’s not 100 percent accurate, but it’s still very impressive.

Another impressive feature is that the list of languages Streamer can translate is over five pages long. That list includes different dialects or accents of the same language; for example, there are 15 versions of Arabic, so you can specify Egyptian versus Algerian, and so on. There are 13 versions of English too, so if you are on an international call and have a hard time deciphering an Irish accent or an Indian one, the real-time captioning can save a lot of time asking people to repeat what they just said. In a military context, operational briefings for multi-lingual forces could take hours to get accurate translations, which is far too long.

All of this functionality was made possible by an SBIR contract—a program that provided more than just money. Robert Palmquist made it very clear that the assistance he got from the Office of Naval Research, specifically Dr. Joel Davis, was essential. “SBIRs give more than funding,” he said. “They give you mentoring and advice.” Through his SBIR contract, Palmquist also found connections to people doing similar work, and he learned how to better understand the military’s needs and how to meet them.