论文标题
3D GAN反转具有姿势优化
3D GAN Inversion with Pose Optimization
论文作者
论文摘要
随着基于NERF的3D Aware Gans质量的最新进展，将图像投射到这些3D感知甘人的潜在空间中比2D GAN倒置具有自然优势：它不仅允许对投影图像进行多视图一致的编辑，而且还允许仅给出单个图像时启用3D重建和新型视图。但是，显式视图控制在3D GAN倒置过程中充当主要的障碍，因为必须同时优化相机姿势和潜在代码才能重建给定的图像。大多数探索3D感知gan的潜在空间的作品都依赖于地面真实的摄像头视图或可变形的3D模型，从而限制了它们的适用性。在这项工作中，我们引入了一种可推广的3D GAN反转方法，该方法同时渗透相机的视点和潜在代码，以启用多视图一致的语义图像编辑。我们方法的关键是利用预训练的估计器来更好地初始化，并利用从NERF参数计算出的像素深度来更好地重建给定的图像。我们对图像重建进行了广泛的实验，并在定量和定性上进行了编辑，并将结果与基于2D GAN的编辑进行了比较，以证明利用3D GAN的潜在空间的优势。可在https://3dgan-inversion.github.io上获得其他结果和可视化。
With the recent advances in NeRF-based 3D aware GANs quality, projecting an image into the latent space of these 3D-aware GANs has a natural advantage over 2D GAN inversion: not only does it allow multi-view consistent editing of the projected image, but it also enables 3D reconstruction and novel view synthesis when given only a single image. However, the explicit viewpoint control acts as a main hindrance in the 3D GAN inversion process, as both camera pose and latent code have to be optimized simultaneously to reconstruct the given image. Most works that explore the latent space of the 3D-aware GANs rely on ground-truth camera viewpoint or deformable 3D model, thus limiting their applicability. In this work, we introduce a generalizable 3D GAN inversion method that infers camera viewpoint and latent code simultaneously to enable multi-view consistent semantic image editing. The key to our approach is to leverage pre-trained estimators for better initialization and utilize the pixel-wise depth calculated from NeRF parameters to better reconstruct the given image. We conduct extensive experiments on image reconstruction and editing both quantitatively and qualitatively, and further compare our results with 2D GAN-based editing to demonstrate the advantages of utilizing the latent space of 3D GANs. Additional results and visualizations are available at https://3dgan-inversion.github.io .
