Gesture-augmented speech recognition

摘要

Methods and systems may provide for generating text based on speech input and recognizing one or more hand gestures. Additionally, an adaptation of the text may be conducted based on the hand gestures. In one example, the hand gestures are associated with operations such as punctuation insertion operations, cursor movement operations, text selection operations, capitalization operations, pause of speech recognition operations, resume of speech recognition operations, application-specific actions, and so forth, wherein the adaptation of the text includes the associated operation.

主权项

1. A speech recognition method comprising:
generating text based on speech input using a speech module; receiving image data that supplements the speech input using a recognition system and one or more of an infrared camera, a time of flight camera and a plurality of stereoscopic cameras; recognizing, by a speech and gesture recognition apparatus, one or more hand gestures based on wearable accelerometer device data using a gesture module, wherein recognizing the one or more hand gestures includes identifying one or more of fingers and a direction of movement; associating, by the speech and gesture recognition apparatus, the one or more hand gestures with an operation using the recognition system and including one or more of a punctuation insertion, a cursor movement, a text selection, a capitalization, a pause of speech recognition, a resume of speech recognition and an application-specific action; and conducting, by the speech and gesture recognition apparatus, an adaptation of the text using an adaptation module, wherein the adaptation includes the operation, wherein the application-specific action includes switching between a value mode and a formula mode in a cell of a spreadsheet application based on the one or more recognized hand gestures.