Multi-view synthesis in real-time with fallback to 2D from 3D to reduce flicker in low or unstable stereo-matching image regions

摘要

Multi view images are generated with reduced flickering. A first depth map is generated from stereo images by stereo-matching. When stereo-matching is poor or varies too much from frame to frame, disparity fallback selects a second depth map that is generated from a single view without stereo-matching, preventing stereo-matching errors from producing visible artifacts or flickering. Flat or textureless regions can use the second depth map, while regions with good stereo-matching use the first depth map. Depth maps are generated with a one-frame delay and buffered. Low-cost temporal coherence reduces costs used for stereo-matching when the pixel location selected as the lowest-cost disparity is within a distance threshold of the same pixel in a last frame. Hybrid view synthesis uses forward mapping for smaller numbers of views, and backward mapping from the forward-mapping results for larger numbers of views. Rotated masks are generated on-the-fly for backward mapping.

主权项

1. A multi-view synthesizer comprising:
a first depth generator that generates a first depth map by stereo-matching a first stereo image to a second stereo image; a consistency checker that generates a stereo-matching rate indicating a quality of the first depth map, the consistency checker comparing matched pixel locations when starting with the first stereo image to pixel locations when starting with the second stereo image to generate the stereo-matching rate; a second depth generator that generates a second depth map from no more than one single view image; a disparity fallback selector that selects the second depth map as a selected depth map instead of the first depth map when the stereo-matching rate is below a matching threshold; a forward mapper that shifts sub-pixels from a source image by an offset to generate a destination image, the offset being read from or generated from the selected depth map, wherein the offset is scaled to a different value for each view image in a plurality of multi view images output as the destination image; and a backward mapper that reads a plurality of view masks to locate a selected view mask that generates a sub-pixel location in the destination image, and locates a source sub-pixel in the source image using a view offset for the selected view mask, the backward mapper generating the destination image by copying the source sub-pixel to the sub-pixel location in the destination image.