Structural Multiplane Image: Bridging Neural View Synthesis and 3D Reconstruction.
2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)(2023)
摘要
The Multiplane Image (MPI), containing a set of fronto-parallel $RGB_{\alpha}$ layers, is an effective and efficient representation for view synthesis from sparse inputs. Yet, its fixed structure limits the performance, especially for surfaces imaged at oblique angles. We introduce the Structural MPI (S-MPI), where the plane structure approximates 3D scenes concisely. Conveying $RGB_{\alpha}$ contexts with geometrically-faithful structures, the S-MPI directly bridges view synthe-sis and 3D reconstruction. It can not only overcome the critical limitations of MPI, i.e., discretization artifacts from sloped surfaces and abuse of redundant layers, and can also acquire planar 3D reconstruction. Despite the intu-ition and demand of applying S-MPI, great challenges are introduced, $e.g$ ., high-fidelity approximation for both $RGB_{\alpha}$ layers and plane poses, multi-view consistency, non-planar regions modeling, and efficient rendering with intersected planes. Accordingly, we propose a transformer-based network based on a segmentation model [4]. It predicts compact and expressive S-MPI layers with their corresponding masks, poses, and $RGB_{\alpha}$ contexts. Non-planar regions are inclusively handled as a special case in our unified frame-work. Multi-view consistency is ensured by sharing global proxy embeddings, which encode plane-level features cov-ering the complete 3D scenes with aligned coordinates. In-tensive experiments show that our method outperforms both previous state-of-the-art MPI-based view synthesis methods and planar reconstruction methods.
更多查看译文
关键词
Image and video synthesis and generation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn