Large Spatial Model：从无位姿图像到语义3D的端到端建模
Large Spatial Model: End-to-end Unposed Images to Semantic 3D
AI总结：
针对传统多步3D重建流程耗时长、复杂度高的问题，提出Large Spatial Model，端到端将无位姿图像转为语义辐射场，实现实时语义3D重建并支持语言驱动的场景操作。
摘要：
从有限数量的图像中重建并理解3D结构是计算机视觉领域的一个经典问题。传统方法通常将该任务拆分为多个子任务，每个子任务都需要在不同数据表示之间进行复杂转换。例如，通过运动恢复结构（Structure-from-Motion, SfM）进行稠密重建时，需要将图像转换为关键点、优化相机参数并估计结构。之后，还需要精确的稀疏重建结果来开展进一步的稠密建模，再将其输入到特定任务的神经网络中。这种多步骤流程会导致处理时间大幅增加，工程复杂度也随之提升。\n在本研究中，我们提出了Large Spatial Model（LSM，大空间模型），它可将无位姿RGB图像直接处理成语义辐射场。LSM通过单次前向运算同时估计几何结构、外观和语义信息，还能在新视角下通过与语言交互生成多样化的标签图。LSM采用基于Transformer的架构，通过像素对齐点图整合全局几何信息。为增强空间属性回归效果，我们引入了带多尺度融合的局部上下文聚合机制，提升了精细局部细节的准确性。为解决带标注3D语义数据稀缺的问题，并实现自然语言驱动的场景操作，我们将预训练的2D语言驱动分割模型整合到3D一致的语义特征场中。随后，一个高效解码器对一组语义各向异性高斯进行参数化，为有监督端到端学习提供了便利。在多个任务上开展的大量实验表明，LSM可直接从无位姿图像出发统一处理多项3D视觉任务，首次实现了实时语义3D重建。
英文摘要：
Reconstructing and understanding 3D structures from a limited number of images is a well-established problem in computer vision. Traditional methods usually break this task into multiple subtasks, each requiring complex transformations between different data representations. For instance, dense reconstruction through Structure-from-Motion (SfM) involves converting images into key points, optimizing camera parameters, and estimating structures. Afterward, accurate sparse reconstructions are required for further dense modeling, which is subsequently fed into task-specific neural networks. This multi-step process results in considerable processing time and increased engineering complexity. In this work, we present the Large Spatial Model (LSM), which processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation, and it can generate versatile label maps by interacting with language at novel viewpoints. Leveraging a Transformer-based architecture, LSM integrates global geometry through pixel-aligned point maps. To enhance spatial attribute regression, we incorporate local context aggregation with multi-scale fusion, improving the accuracy of fine local details. To tackle the scarcity of labeled 3D semantic data and enable natural language-driven scene manipulation, we incorporate a pre-trained 2D language-based segmentation model into a 3D-consistent semantic feature field. An efficient decoder then parameterizes a set of semantic anisotropic Gaussians, facilitating supervised end-to-end learning. Extensive experiments across various tasks show that LSM unifies multiple 3D vision tasks directly from unposed images, achieving real-time semantic 3D reconstruction for the first time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
