Determining feature vectors for video volumes

摘要

A volume identification system identifies a set of unlabeled spatio-temporal volumes within each of a set of videos, each volume representing a distinct object or action. The volume identification system further determines, for each of the videos, a set of volume-level features characterizing the volume as a whole. In one embodiment, the features are based on a codebook and describe the temporal and spatial relationships of different codebook entries of the volume. The volume identification system uses the volume-level features, in conjunction with existing labels assigned to the videos as a whole, to label with high confidence some subset of the identified volumes, e.g., by employing consistency learning or training and application of weak volume classifiers. The labeled volumes may be used for a number of applications, such as training strong volume classifiers, improving video search (including locating individual volumes), and creating composite videos based on identified volumes.

主权项

1. A computer-implemented method comprising:
accessing a feature codebook comprising a set of representative feature vectors representing at least visual properties of digital videos; identifying, in a plurality of digital videos, a plurality of candidate volumes representing spatio-temporal portions of the digital videos, wherein each of the candidate volumes corresponds to a contiguous sequence of spatial portions of video frames having a starting time and an ending time; associating features with each candidate volume of a plurality of the identified candidate volumes, the associating comprising:
identifying a plurality of temporal segments of the candidate volume;for each of the identified temporal segments:
determining a feature vector from at least visual properties of the temporal segment, andassociating with the temporal segment a representative feature vector from the feature codebook that is most similar to the feature vector;determining features for the candidate volume, the features comprising temporal relationship features comprising, for each of a plurality of the representative feature vectors of the feature codebook, quantifications of occurrences of the representative feature vector within the candidate volume with respect to occurrences of other ones of the representative feature vectors within the candidate volume, the occurrences quantified according to a temporal operator;assigning a label to the candidate volume using the determined temporal relationship features, the label indicating a particular object or action represented by the candidate volume; andstoring the label in association with the candidate volume.