Lidar stereo fusion live action 3D model video reconstruction for six degrees of freedom 360° volumetric virtual reality video

摘要

A system for capturing live-action three-dimensional video is disclosed. The system includes pairs of stereo cameras and a LIDAR for generating stereo images and three-dimensional LIDAR data from which three-dimensional data may be derived. A depth-from-stereo algorithm may be used to generate the three-dimensional camera data for the three-dimensional space from the stereo images and may be combined with the three-dimensional LIDAR data taking precedence over the three-dimensional camera data to thereby generate three-dimensional data corresponding to the three-dimensional space.

主权项

1. A system for capturing live action three-dimensional video comprising:
a first and a second camera, making up a stereo camera pair, the first and second stereo cameras each having known imaging parameters such that a depth-from-stereo algorithm may be applied to corresponding sets of images captured using the stereo camera pair to generate three-dimensional camera data for a three-dimensional space viewed by the stereo camera pair; a LIDAR, having a known position relative to the stereo camera pair, configured to produce three-dimensional LIDAR data; a time-stamp device for applying a first time-stamp to the three-dimensional camera data and a second time-stamp to the three-dimensional LIDAR data; and a controller configured to
use the depth-from-stereo algorithm to generate the three-dimensional camera data for the three-dimensional space;combine the three-dimensional camera data and the three-dimensional LIDAR data, with the three-dimensional LIDAR data taking precedence over the three-dimensional camera data to thereby generate three-dimensional data corresponding to the three-dimensional space;detect image seams between each of the sets of images using one of color matching and gradient detection;generate a low resolution version of each of the sets of images;select preliminary blend colors for each pixel on the image seams of the low resolution version of each of the sets of images;provide the preliminary blend colors as a baseline for use in selecting blend colors for use in performing color blending for the sets of images at captured resolution to thereby lower a search space for appropriate blend colors for the image seams between the sets of images at the captured resolution;select a blend color for each pixel along the image seams using the preliminary blend color that corresponds to the associated pixel at the captured resolution as a starting point for a search for an appropriate blend color for each pixel of the sets of images at the captured resolution; andperform color blending on the image seams using the selected blend color for each pixel.