System and method for using information from intuitive multimodal interactions for media tagging

摘要

System and method for using information extracted from intuitive multimodal interactions in the context of media for media tagging are disclosed. In one embodiment, multimodal information related to media is captured during multimodal interactions of a plurality of users. The multimodal information includes speech information and gesture information. Further, the multimodal information is analyzed to identify speech portions of interest. Furthermore, relevant tags for tagging the media are extracted from the speech portions of interest.

主权项

1. A method of using multimodal information for media tagging, comprising:
capturing, during multimodal interactions of a plurality of users, multimodal information related to media, wherein the multimodal information comprises speech information and gesture information of the plurality of users; identifying an occurrence of a pre-determined keyword in the speech information; identifying a co-occurrence of a pre-determined gesture from the gesture information with the occurrence of the pre-determined keyword in the speech information; identifying a speech portion of interest in the speech information corresponding to the identified co-occurrence of the pre-determined gesture and the pre-determined keyword, wherein the speech portion of interest includes speech for a specified time duration before and after the occurrence of the identified keyword; and tagging the media by attaching the identified speech portion of interest to the media.