Automatic geometry and lighting inference for realistic image editing

摘要

Image editing techniques are disclosed that support a number of physically-based image editing tasks, including object insertion and relighting. The techniques can be implemented, for example in an image editing application that is executable on a computing system. In one such embodiment, the editing application is configured to compute a scene from a single image, by automatically estimating dense depth and diffuse reflectance, which respectively form the geometry and surface materials of the scene. Sources of illumination are then inferred, conditioned on the estimated scene geometry and surface materials and without any user input, to form a complete 3D physical scene model corresponding to the image. The scene model may include estimates of the geometry, illumination, and material properties represented in the scene, and various camera parameters. Using this scene model, objects can be readily inserted and composited into the input image with realistic lighting, shadowing, and perspective.

主权项

1. A method for automatically estimating illumination sources associated with a digital image depicting a scene, the method comprising:
detecting light sources depicted in the digital image that meet a given thresholding requirement, the digital image comprising a plurality of pixels; estimating a dense depth associated with the digital image by
estimating a focal length of a camera used to capture the digital image and a sparse surface orientation map based on geometric constraints depicted in the digital image,applying a non-parametric depth sampling approach that uses a dataset of RGB-D images to estimate a depth at every pixel of the digital image that is consistent with the sparse surface orientation map, andproviding a set of estimated depths and normals that represent a geometry model of the scene, wherein geometric cues provided by the sparse surface orientation map are used in estimating the dense depth to enforce orientation constraints, piecewise-planarity, and surface smoothness; pruning the detected light sources using the estimated dense depth; and estimating light outside of a view frustum associated with the digital image using a dataset of image-based lights comprising spherical high dynamic range images, wherein each spherical image is sub-sampled into rectilinear projections and matched to the digital image, and wherein one or more top matching candidate image-based lights are used as distinct sources of light.