SYSTEMS AND METHODS FOR NATURAL INTERACTION WITH OPERATING SYSTEMS AND APPLICATION GRAPHICAL USER INTERFACES USING GESTURAL AND VOCAL INPUT

摘要

Systems and methods for natural interaction with graphical user interfaces using gestural and vocal input in accordance with embodiments of the invention are disclosed. In one embodiment, a method for interpreting a command sequence that includes a gesture and a voice cue to issue an application command includes receiving image data, receiving an audio signal, selecting an application command from a command dictionary based upon a gesture identified using the image data, a voice cue identified using the audio signal, and metadata describing combinations of a gesture and a voice cue that form a command sequence corresponding to an application command, retrieving a list of processes running on an operating system, selecting at least one process based upon the selected application command and the metadata, where the metadata also includes information identifying at least one process targeted by the application command, and issuing an application command to the selected process.

主权项

1. A method for interpreting a command sequence that includes a gesture and a voice cue to issue an application command to at least one process using a natural interaction user interface system that includes a processor and memory containing a command dictionary, the method comprising:
receiving image data using a natural interaction user interface system; receiving an audio signal using a natural interaction user interface system; selecting, using a natural interaction user interface system, an application command from a command dictionary of application commands based upon:
a gesture identified using the image data;a voice cue identified using the audio signal; andmetadata describing combinations of a gesture and a voice cue that form a command sequence corresponding to an application command within the command dictionary;retrieving a list of processes running on an operating system using a natural interaction user interface system;selecting at least one process from the list of processes based upon the selected application command and the metadata using a natural interaction user interface system, where the metadata further comprises information identifying at least one process targeted by the application command; andissuing an application command to the selected at least one process using a natural interaction user interface system.