Methods and apparatus for video completion

摘要

Methods, apparatus, and computer-readable storage media for video completion that may be applied to restore missing content, for example holes or border regions, in video sequences. A video completion technique applies a subspace constraint technique that finds and tracks feature points in the video, which are used to form a model of the camera motion and to predict locations of background scene points in frames where the background is occluded. Another frame where those points were visible is found, and that frame is warped using the predicted points. A content-preserving warp technique may be used. Image consistency constraints may be applied to modify the warp so that it fills the hole seamlessly. A compositing technique is applied to composite the warped image into the hole. This process may be repeated until the missing content is filled on all frames.

主权项

1. A method, comprising:
obtaining an input video sequence of a scene comprising a plurality of frames and an indication of an object to be removed from the frames of the input video sequence; tracking feature points in the input video sequence to generate feature tracks; factoring the feature tracks into a low-dimensional subspace; generating a prediction of background scene motion according to the low-dimensional subspace; and processing a frame of the input video sequence designated as a target frame to fill a region in the target frame occluded by the indicated object with background content from unoccluded regions of one or more other frames of the input video sequence designated as source frames according to the prediction of background scene motion, the occluded region in the target frame is filled using one said unoccluded region from one said source frame at a time such that:
if the one said unoccluded region from the one said source frame is determined to cause an entirety of the occluded region to be filled then compositing the one said source frame to fill the occluded region; orif the one said unoccluded region from the one said source frame is determined not to cause the entirety of the occluded region to be filled, then compositing the one said source frame to fill any portion of the occluded region that the background content of the one said source frame is determined to fill, and selecting a next said source frame to fill at least part of a remaining portion of the occluded region.