UniSem: 从稀疏未定位图像进行通用语义3D重建
UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images
AI总结：
UniSem通过误差感知高斯丢弃和混合训练课程提升深度精度和语义泛化能力，在ScanNet和Replica数据集上实现了更优的深度预测和开放词汇3D分割性能。
摘要：
语义感知的3D重建从稀疏、未定位图像仍对前馈3D高斯点绘制（3DGS）具有挑战性。现有方法在稀疏视图监督下往往预测过多的高斯基础元素，导致几何不稳定和较差的深度质量。同时，它们仅依赖2D分割器特征进行语义提升，提供弱3D级和有限可泛化的监督，导致新场景中的语义不完整。为了解决这些问题，我们提出了UniSem，一个统一框架，通过两个关键组件联合提升深度精度和语义泛化能力。首先，误差感知高斯丢弃（EGD）通过渲染误差提示抑制易冗余的高斯，产生有意义且几何稳定的高斯表示以提高深度估计。其次，我们引入混合训练课程（MTC），逐步融合2D分割器提升的语义与模型自身涌现的3D语义先验，通过对象级原型对齐来增强语义一致性和完整性。在ScanNet和Replica上进行的大量实验表明，UniSem在不同输入视图数量下实现了更优的深度预测和开放词汇3D分割性能。值得注意的是，使用16视图输入时，UniSem将深度相对误差（Rel）减少了15.2%，并在开放词汇分割中将mAcc提高了3.7%。
英文摘要：
Semantic-aware 3D reconstruction from sparse, unposed images remains challenging for feed-forward 3D Gaussian Splatting (3DGS). Existing methods often predict an over-complete set of Gaussian primitives under sparse-view supervision, leading to unstable geometry and inferior depth quality. Meanwhile, they rely solely on 2D segmenter features for semantic lifting, which provides weak 3D-level and limited generalizable supervision, resulting in incomplete 3D semantics in novel scenes. To address these issues, we propose UniSem, a unified framework that jointly improves depth accuracy and semantic generalization via two key components. First, Error-aware Gaussian Dropout (EGD) performs error-guided capacity control by suppressing redundancy-prone Gaussians using rendering error cues, producing meaningful, geometrically stable Gaussian representations for improved depth estimation. Second, we introduce a Mix-training Curriculum (MTC) that progressively blends 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors, implemented with object-level prototype alignment to enhance semantic coherence and completeness. Extensive experiments on ScanNet and Replica show that UniSem achieves superior performance in depth prediction and open-vocabulary 3D segmentation across varying numbers of input views. Notably, with 16-view inputs, UniSem reduces depth Rel by 15.2% and improves open-vocabulary segmentation mAcc by 3.7% over strong baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
