Systems and methods for implementing and using gesture based user interface widgets with camera input

摘要

Described is an approach to gesture interaction that is based on user interface widgets. In order to detect user gestures, the widgets are provided with hotspots that are monitored using a camera for predetermined patterns of occlusion. A hotspot is a region where a user interacts with the widget by making a gesture over it. The user's gesture may be detected without the user physically touching the surface displaying the widget. The aforesaid hotspots are designed to be visually salient and suggestive of the type of gestures that can be received from the user. Described techniques are advantageous in relation to conventional systems, such as systems utilizing finger tracking, in that they can better support complex tasks with repeated user actions. In addition, they provide better perceived affordance than conventional systems that attempt to use widgets that are not designed for gesture input, or in-the-air gesture detection techniques that lack any visual cues.

主权项

1. A computer-implemented method for detecting a gesture performed by a user in connection with a user interface widget comprising a hotspot, the method being performed in a computerized system comprising a central processing unit, a camera and a memory, the computer-implemented method comprising:
a. acquiring a plurality of images of the hotspot using the camera; b. using the central processing unit to identify occlusion of at least a portion of the hotspot by the user in the acquired plurality of images of the hotspot, wherein the occlusion of the portion of the hotspot is identified by detecting changes in color values of each of a plurality of non-contiguous image sensor pixels within the acquired plurality of images of the hotspot based on a recorded history of values of the plurality of non-contiguous image sensor pixels; and c. using the central processing unit to detect the gesture performed by the user based on the identified occlusion of the at least the portion of the hotspot in at least two of the acquired plurality of images of the hotspot.