Method and apparatus for virtual viewpoint synthesis in multi-viewpoint video

摘要

Disclosed are a virtual viewpoint combination method and device in a multi-viewpoint video, improving the quality of the virtual viewpoint images and improving the display effects of the virtual viewpoint images under the premise of not introducing geometrical distortion. The method includes: taking two viewpoint images provided by a first camera and a second camera as reference viewpoint images, and respectively performing 3D image conversion, pinhole filling, image compensation and image convergence processing on the two viewpoint images to obtain a main virtual viewpoint image; then respectively performing depth-image processing, 3D image conversion, pinhole filling, image compensation and image convergence processing on the two viewpoint images to obtain a subordinate virtual viewpoint image; and performing filling the remaining holes in the main virtual viewpoint image on the basis of the subordinate virtual viewpoint image to obtain a final virtual viewpoint image to be combine.

主权项

1. A method for virtual viewpoint synthesis in a multi-viewpoint video, comprising:
the step 1 of performing three-dimension image transformation on a first reference viewpoint image obtained by a first camera according to the first reference viewpoint image and its depth image, and parameters of the first camera, to obtain a first virtual viewpoint image and its depth image and hole map; and performing three-dimension image transformation on a second reference viewpoint image obtained by a second camera according to the second reference viewpoint image and its depth image, and parameters of the second camera, to obtain a second virtual viewpoint image and its depth image and hole map; the step 2 of filling small holes of the first virtual viewpoint image through horizontal, vertical and diagonal extrapolations according to the hole map of the first virtual viewpoint image, to obtain the small holes-filled first virtual viewpoint image; and filling small holes of the second virtual viewpoint image through horizontal, vertical and diagonal extrapolations according to the hole map of the second virtual viewpoint image, to obtain the small holes-filled second virtual viewpoint image; the step 3 of performing image complementing by taking the small holes-filled first virtual viewpoint image and second virtual viewpoint image as reference images of each other, to obtain the image-complemented first virtual viewpoint image and second virtual viewpoint image; the step 4 of performing fusion processing on the image-complemented first virtual viewpoint image and second virtual viewpoint image, to obtain a primary virtual viewpoint image; the step 5 of performing depth image processing respectively on the first reference viewpoint image and the second reference viewpoint image, to obtain the depth image-processed first reference viewpoint image and second reference viewpoint image; and then performing the corresponding processes on the depth image-processed first reference viewpoint image and second reference viewpoint image according to the step 1 to the step 4, to obtain a secondary virtual viewpoint image; and the step 6 of filling holes of the primary virtual viewpoint image according to the secondary virtual viewpoint image to generate a virtual viewpoint image.