论文标题
3D场景操纵的体积拆卸
Volumetric Disentanglement for 3D Scene Manipulation
论文作者
论文摘要
最近，差异化渲染的进步使复杂的3D场景的照片现实和精细详细重建可以重大突破，这对于许多虚拟现实应用程序都是关键。但是，在增强现实的背景下，人们可能还希望实现场景中对象的语义操纵或增强。为此，我们为（i）解开或分离提出了一个体积框架，给定前景对象与背景的体积表示，以及（ii）在语义上操纵前景对象以及背景。我们的框架作为输入一组2D掩模，指定所需的前景对象，以及相关的2D视图和姿势，并产生一个前景背景的分离，以尊重周围的照明，反射和部分遮挡，可以将其应用于训练和新型视图。我们的方法可以单独控制像素颜色和深度以及前景对象和背景对象的3D相似性转换。随后，我们演示了我们的框架在许多下游操作任务上的适用性，包括对象伪装，非阴性3D对象介入，3D对象翻译，3D对象介入和基于3D文本的对象操作。在我们的项目网页上给出了完整的结果
Recently, advances in differential volumetric rendering enabled significant breakthroughs in the photo-realistic and fine-detailed reconstruction of complex 3D scenes, which is key for many virtual reality applications. However, in the context of augmented reality, one may also wish to effect semantic manipulations or augmentations of objects within a scene. To this end, we propose a volumetric framework for (i) disentangling or separating, the volumetric representation of a given foreground object from the background, and (ii) semantically manipulating the foreground object, as well as the background. Our framework takes as input a set of 2D masks specifying the desired foreground object for training views, together with the associated 2D views and poses, and produces a foreground-background disentanglement that respects the surrounding illumination, reflections, and partial occlusions, which can be applied to both training and novel views. Our method enables the separate control of pixel color and depth as well as 3D similarity transformations of both the foreground and background objects. We subsequently demonstrate the applicability of our framework on a number of downstream manipulation tasks including object camouflage, non-negative 3D object inpainting, 3D object translation, 3D object inpainting, and 3D text-based object manipulation. Full results are given in our project webpage at https://sagiebenaim.github.io/volumetric-disentanglement/
