先造假后成真：利用虚拟多视图增强单目室内语义场景补全
Fake It To Make It: Virtual Multiviews to Enhance Monocular Indoor Semantic Scene Completion
AI总结：
本文提出GenFuSE框架，通过虚拟多视图合成与多视图融合适配器增强单目室内语义场景补全，在NYUv2上最高提升4.9%的语义IoU。
摘要：
单目室内语义场景补全（SSC）旨在从室内场景的单张RGB图像重建三维语义占用图，从二维图像线索推断空间布局和物体类别。该任务的挑战源于将二维图像转换到三维空间时产生的深度、尺度和形状歧义，尤其是在复杂且通常严重遮挡的室内环境中。当前的SSC方法常常难以应对这些歧义，导致物体表示发生畸变或缺失。为克服这些局限，我们提出一种创新方法，利用新视角合成和多视图融合。具体而言，我们展示了如何在场景周围放置虚拟相机，以模拟多视图输入，从而增强上下文场景信息。我们还引入了一个多视图融合适配器（MVFA），将多视图三维场景预测有效融合为统一的三维语义占用图。最后，我们识别并研究了生成式技术应用于SSC时的固有限制，即新颖性-一致性权衡。我们的系统GenFuSE在与NYUv2数据集上的现有SSC网络集成时，场景补全的IoU分数提升最高达2.8%，语义场景补全提升最高达4.9%。本工作将GenFuSE作为一个标准框架，用于通过合成输入推进单目SSC。
英文摘要：
Monocular Indoor Semantic Scene Completion (SSC) aims to reconstruct a 3D semantic occupancy map from a single RGB image of an indoor scene, inferring spatial layout and object categories from 2D image cues. The challenge of this task arises from the depth, scale, and shape ambiguities that emerge when transforming a 2D image into 3D space, particularly within the complex and often heavily occluded environments of indoor scenes. Current SSC methods often struggle with these ambiguities, resulting in distorted or missing object representations. To overcome these limitations, we introduce an innovative approach that leverages novel view synthesis and multiview fusion. Specifically, we demonstrate how virtual cameras can be placed around the scene to emulate multiview inputs that enhance contextual scene information. We also introduce a Multiview Fusion Adaptor (MVFA) to effectively combine the multiview 3D scene predictions into a unified 3D semantic occupancy map. Finally, we identify and study the inherent limitation of generative techniques when applied to SSC, specifically the Novelty-Consistency tradeoff. Our system, GenFuSE, demonstrates IoU score improvements of up to 2.8% for Scene Completion and 4.9% for Semantic Scene Completion when integrated with existing SSC networks on the NYUv2 dataset. This work introduces GenFuSE as a standard framework for advancing monocular SSC with synthesized inputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
