AUDIO-VISUAL INTERACTION WITH USER DEVICES

摘要

A user device is enabled by an audio-visual assistant for audio-visual interaction with a user. The audio-visual assistant enables the user device to track the user's eyes and face to determine objects on the screen that the user is currently observing. Various tasks can be executed on the objects based on further input provided by the user. The user can provide further inputs via facial gestures, voice or combinations thereof for executing the various tasks.

主权项

1) A method comprising:
accessing, by a processor, an audio-visual assistant that is executing on a user device, the audio-visual assistant comprising an optical detection function for detecting eye movements of a user of the device and an audio detection function that enables audio interaction by the user with the device; determining, by the processor via the audio-visual assistant, a portion of a display screen currently being observed by the user; identifying, by the processor, a user interface element displayed within the portion of the display screen; receiving, by the processor, a command from the user to control the identified user interface element; selecting, by the processor, a task to be associated with the user interface element from a plurality of tasks, the plurality of tasks corresponding to respective plurality of commands associated with the audio visual assistant; and executing, by the processor, the selected task associated with the user interface element based at least on the received command from the user.