arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2412.12906cs.CV

CATSplat:用于从单视图图像进行可泛化3D高斯泼溅的带空间引导的上下文感知Transformer

CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Image

Korea University(高丽大学);Google(谷歌公司);Purdue University(普渡大学)

Wonseok Roh, Hwanhee Jung, Jong Wook Kim, Seunggwan Lee, Innfarn Yoo, Andreas Lugmayr, Seunggeun Chi, Karthik Ramani, Sangpil Kim

AI总结:

提出CATSplat框架,通过引入视觉语言模型的文本指导和3D点特征的空间指导,突破单目设置限制,实现高质量的单视图3D场景重建与新视图合成。

摘要:

最近,基于3D Gaussian Splatting的可泛化前馈方法因其在使用有限资源重建3D场景方面的潜力而受到广泛关注。这些方法通过单次前向传播,仅从少量图像中创建由逐像素3D高斯原语参数化的3D辐射场。然而,与受益于跨视图对应关系的多视图方法不同,使用单视图图像进行3D场景重建仍然是一个尚未充分探索的领域。在这项工作中,我们引入了CATSplat,一种新颖的基于Transformer的可泛化框架,旨在突破单目设置中的固有约束。首先,我们提出利用来自视觉语言模型的文本指导,以补充单张图像信息的不足。通过交叉注意力从文本嵌入中引入场景特定的上下文细节,我们为实现超越仅依赖视觉线索的上下文感知3D场景重建铺平了道路。此外,我们主张在单视图设置下利用来自3D点特征的空间指导,以实现全面的几何理解。借助3D先验,图像特征能够捕获丰富的结构信息,从而在无需多视图技术的情况下预测3D高斯。在大规模数据集上的大量实验表明,CATSplat在单视图3D场景重建及高质量新视图合成方面达到了最先进的性能。

英文摘要:

Recently, generalizable feed-forward methods based on 3D Gaussian Splatting have gained significant attention for their potential to reconstruct 3D scenes using finite resources. These approaches create a 3D radiance field, parameterized by per-pixel 3D Gaussian primitives, from just a few images in a single forward pass. However, unlike multi-view methods that benefit from cross-view correspondences, 3D scene reconstruction with a single-view image remains an underexplored area. In this work, we introduce CATSplat, a novel generalizable transformer-based framework designed to break through the inherent constraints in monocular settings. First, we propose leveraging textual guidance from a visual-language model to complement insufficient information from a single image. By incorporating scene-specific contextual details from text embeddings through cross-attention, we pave the way for context-aware 3D scene reconstruction beyond relying solely on visual cues. Moreover, we advocate utilizing spatial guidance from 3D point features toward comprehensive geometric understanding under single-view settings. With 3D priors, image features can capture rich structural insights for predicting 3D Gaussians without multi-view techniques. Extensive experiments on large-scale datasets demonstrate the state-of-the-art performance of CATSplat in single-view 3D scene reconstruction with high-quality novel view synthesis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑