Virtual viewpoint image synthesizing method and virtual viewpoint image synthesizing system

摘要

Provided is a virtual viewpoint image synthesizing method in which a virtual viewpoint image viewed from a virtual viewpoint is synthesized based on image information obtained from a plurality of viewpoints. The virtual viewpoint image is synthesized through a reference images obtaining step, a depth maps generating step, an up-sampling step, a virtual viewpoint information obtaining step, and a virtual viewpoint image synthesizing step.

主权项

1. A virtual viewpoint image synthesizing method in which a virtual viewpoint image viewed from a virtual viewpoint is synthesized based on image information obtained from a plurality of viewpoints, the method comprising:
a reference images obtaining step of obtaining reference images, which become references for the virtual viewpoint image, from a plurality of image obtaining devices disposed at the plurality of viewpoints; a depth maps generating step of generating depth maps of images at the viewpoints at which the plurality of image obtaining devices are disposed by an image depths obtaining device that obtains depths of the images at the viewpoints at which the plurality of image obtaining devices are disposed; an up-sampling step of up-sampling the depth maps generated in the depth maps generating step, and the up-sampling step comprising the steps of:
inputting the depth map input from a depth camera;associating a set of neighboring pixels in the depth map generated in the depth maps generating step with pixels not neighboring each other in the reference image;assigning a weight to each pixel in the set of neighboring pixels in the depth map;optimizing the weight assigned to each pixel in the set of neighboring pixels;calculating a minimum weight; andselecting an optimal depth value for the set of neighboring pixels; a virtual viewpoint information obtaining step of obtaining location information and direction information of the virtual viewpoint from a virtual viewpoint information obtaining device, which obtains the location information and the direction information of the virtual viewpoint, the direction information including a direction in which the synthesized image is viewed from the virtual viewpoint; and a virtual viewpoint image synthesizing step of synthesizing the virtual viewpoint image, which corresponds to the location information and the direction information of the virtual viewpoint obtained in the virtual viewpoint information obtaining step, based on the reference images obtained in the reference images obtaining step, the depth maps up-sampled in the up-sampling step, and the location information and the direction information.