arXiv 2504.095882025-08-21cs.CVcs.AI

TextSplat：面向可泛化高斯泼溅的文本引导语义融合

TextSplat: Text-Guided Semantic Fusion for Generalizable Gaussian Splatting

Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究所）；Guangdong Laboratory of Artificial Intelligence（广东省人工智能与数字经济实验室）；Peking University（北京大学）；School of Future Technology, South China University of Technology（华南理工大学未来技术学院）；Hangzhou Dianzi University（杭州电子科技大学）；Central Laboratory of Lishui Hospital of Wenzhou Medical University, The First Affiliated Hospital of Lishui University, Lishui People's Hospital（丽水市人民医院中央实验室、丽水大学第一附属医院、丽水人民医院）

Zhicong Wu, Hongbin Xu, Gang Xu, Ping Nie, Zhixin Yan, Jinkai Zheng, Liangqiong Qu, Ming Li, Liqiang Nie

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出首个文本驱动的可泛化高斯泼溅框架TextSplat，通过文本引导融合深度、语义与多视图特征，提升三维重建的几何与语义对齐质量。

摘要：

可泛化高斯泼溅（Generalizable Gaussian Splatting）的最新进展通过利用前馈高斯泼溅模型，实现了从稀疏输入视图进行稳健的三维重建，取得了优越的跨场景泛化能力。然而，许多方法虽然关注几何一致性，却往往忽视了文本驱动引导在增强语义理解方面的潜力，而语义理解对于准确重建复杂场景中的细粒度细节至关重要。为了解决这一局限，我们提出了 TextSplat——首个文本驱动的可泛化高斯泼溅框架。通过采用文本引导的多样化语义线索融合，我们的框架学习稳健的跨模态特征表示，从而改善几何信息与语义信息的对齐，生成高保真的三维重建结果。具体而言，我们的框架采用三个并行模块来获取互补表示：扩散先验深度估计器（Diffusion Prior Depth Estimator）用于获取准确的深度信息，语义感知分割网络（Semantic Aware Segmentation Network）用于获取详细的语义信息，多视图交互网络（Multi-View Interaction Network）用于获取精细化的跨视图特征。随后，在文本引导语义融合模块（Text-Guided Semantic Fusion Module）中，这些表示通过文本引导且基于注意力的特征聚合机制进行整合，从而得到增强的三维高斯参数，并融入详细的语义线索。在多个基准数据集上的实验结果表明，与现有方法相比，本框架在多个评估指标上均取得了性能提升，验证了其有效性。代码将公开发布。

英文摘要：

Recent advancements in Generalizable Gaussian Splatting have enabled robust 3D reconstruction from sparse input views by utilizing feed-forward Gaussian Splatting models, achieving superior cross-scene generalization. However, while many methods focus on geometric consistency, they often neglect the potential of text-driven guidance to enhance semantic understanding, which is crucial for accurately reconstructing fine-grained details in complex scenes. To address this limitation, we propose TextSplat--the first text-driven Generalizable Gaussian Splatting framework. By employing a text-guided fusion of diverse semantic cues, our framework learns robust cross-modal feature representations that improve the alignment of geometric and semantic information, producing high-fidelity 3D reconstructions. Specifically, our framework employs three parallel modules to obtain complementary representations: the Diffusion Prior Depth Estimator for accurate depth information, the Semantic Aware Segmentation Network for detailed semantic information, and the Multi-View Interaction Network for refined cross-view features. Then, in the Text-Guided Semantic Fusion Module, these representations are integrated via the text-guided and attention-based feature aggregation mechanism, resulting in enhanced 3D Gaussian parameters enriched with detailed semantic cues. Experimental results on various benchmark datasets demonstrate improved performance compared to existing methods across multiple evaluation metrics, validating the effectiveness of our framework. The code will be publicly available.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。