Multimodal user interface

摘要

When using finite-state methods to capture multimodal integration, it is desirable to abstract over certain aspects of the gestural content. For abstraction, the gestural input is represented as a transducer which relates the specific content of the gestures to the gesture symbols. After multimodal integration, a projection is taken of the resulting finite state machine and it is composed with the transducer representing the gestural input in order to reincorporate the specific contents that were left out of the finite-state process.