MULTI-MODAL INFORMATION INTEGRATED ANALYSIS DEVICE

摘要

PROBLEM TO BE SOLVED: To integrally analyze voice sounded by a human being and the gesture of the human being and to output an analyzed result. SOLUTION: A voice recognition part 11 voice-recognizes sounded voice and outputs a voice recognized result and the time corresponding to it and a language analysis part 12 performs language analysis by using knowledge relating to languages and outputs the semantic structure of the voice recognized result and the corresponding time. A GUI control part 13 outputs the position on a screen of the track of an inputted gesture and the corresponding time and a gesture analysis part 14 performs analysis by using the knowledge relating to a diagram provided with plural instruction object candidates for information from the GUI control part 13 and outputs the kind of the gesture, the time corresponding to it and an instruction object which is the instruction object candidate instructed by the gesture. An integrated analysis part 15 detects the timewise relation of a retrieved word or phrase corresponding to the gesture and the instruction object and generates the semantic structure for which the semantic structure of the voice recognized result and the semantic structure of the kind of the gesture are integrated.