Enhanced max margin learning on multimodal data mining in a multimedia database

摘要

Multimodal data mining in a multimedia database is addressed as a structured prediction problem, wherein mapping from input to the structured and interdependent output variables is learned. A system and method for multimodal data mining is provided, comprising defining a multimodal data set comprising image information; representing image information of a data object as a set of feature vectors in a feature space; clustering in the feature space to group similar features; associating a non-image representation with a respective image data object based on the clustering; determining a joint feature representation of a respective data object as a mathematical weighted combination of a set of components of the joint feature representation; optimizing a weighting for a plurality of components of the mathematical weighted combination with respect to a prediction error between a predicted classification and a training classification; and employing the mathematical weighted combination for automatically classifying a new data object.

主权项

1. A data mining method, comprising:
receiving a set of multimodal data objects comprising semantically interrelated information of a first type and a second type, each being of a different type selected from the group consisting of image information, audio information, video information, and semantic information; representing at least the first type of information of the multimodal data objects as feature vectors within a feature space comprising the first type of information and the second type of information, and the semantic interrelation between the first type of information and the second type of information; clustering the feature vectors into classified clusters according to at least one semantic clustering criterion by at least one automated processor, to thereby determine a classification of the respective feature vectors; associating data objects with respective members of the set of multimodal data objects by the at least one automated processor, based on the clustering, the associated data objects comprising information of a third type semantically interrelated to the second type of information, selected from the group consisting of images, audio, video and semantic information, wherein the type of information of the third type is distinct from the type of information of the first type; estimating a joint feature representation of the set of multimodal data objects and the associated data objects by the at least one automated processor; optimizing the joint feature representation by the at least one automated processor to provide a structured output space of interdependent objects, based on at least a prediction error criterion, by iteratively solving a dual problem by selectively partitioning data objects into a working set and a non-working set, comprising:
moving the data objects in the non-working set that can be moved without changing an objective function to the working set, and moving the data objects in the working set that can be moved with a decrease in the objective function to the non-working set; receiving a query represented according to the first type of information; and identifying data objects from the set of multimodal data objects that correspond to the query by the at least one automated processor, based on at least the structured output space of interdependent multimodal objects.