Mobile multimodal user interface combining 3D graphics, location-sensitive speech interaction and tracking technologies

摘要

A mobile reality apparatus, system and method for navigating a site are provided. The method includes the steps of determining a location of a user by receiving a location signal from a location-dependent device; loading and displaying a 3D scene of the determined location; determining an orientation of the user; adjusting a viewpoint of the 3D scene by the determined orientation; determining if the user is within a predetermined distance of an object of interest; and loading a speech dialog of the object of interest. The system includes a plurality of location-dependent devices for transmitting a signal indicative of each devices' location; and a navigation device including a tracking component for determining a position and orientation of the user; a graphic management component for displaying scenes of the site to the user on a display; and a speech interaction component for instructing the user.