TIGaussian: 解耦高斯分布以实现空间感知的图文-3D对齐
TIGaussian: Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment
AI总结：
TIGaussian通过多分支3DGS分词器和模态特定对齐策略，实现空间感知的图文-3D跨模态对齐，提升3D相关任务的预训练效果。
摘要：
尽管视觉-语言模型已经深刻地将文本和图像之间的特征联系起来，但将3D模态数据（如点云和3D高斯分布）纳入其中，进一步使预训练用于3D相关任务成为可能，例如跨模态检索、零样本分类和场景识别。由于在提取3D模态特征和弥合不同模态之间差距方面仍存在挑战，我们提出了TIGaussian，一个利用3D高斯散射（3DGS）特性，通过多分支3DGS分词器和模态特定的3D特征对齐策略来加强跨模态对齐的框架。具体而言，我们的多分支3DGS分词器将3DGS结构的内在属性解耦为紧凑的潜在表示，使特征提取更加通用化。为进一步弥合模态差距，我们开发了双向跨模态对齐策略：一个利用扩散先验的多视图特征融合机制，用于解决图像-3D对齐中的视角模糊问题，同时一个文本-3D投影模块会自适应地将3D特征映射到文本嵌入空间，以实现更好的文本-3D对齐。在各种数据集上的广泛实验表明，TIGaussian在多个任务中表现出最先进的性能。
英文摘要：
While visual-language models have profoundly linked features between texts and images, the incorporation of 3D modality data, such as point clouds and 3D Gaussians, further enables pretraining for 3D-related tasks, e.g., cross-modal retrieval, zero-shot classification, and scene recognition. As challenges remain in extracting 3D modal features and bridging the gap between different modalities, we propose TIGaussian, a framework that harnesses 3D Gaussian Splatting (3DGS) characteristics to strengthen cross-modality alignment through multi-branch 3DGS tokenizer and modality-specific 3D feature alignment strategies. Specifically, our multi-branch 3DGS tokenizer decouples the intrinsic properties of 3DGS structures into compact latent representations, enabling more generalizable feature extraction. To further bridge the modality gap, we develop a bidirectional cross-modal alignment strategies: a multi-view feature fusion mechanism that leverages diffusion priors to resolve perspective ambiguity in image-3D alignment, while a text-3D projection module adaptively maps 3D features to text embedding space for better text-3D alignment. Extensive experiments on various datasets demonstrate the state-of-the-art performance of TIGaussian in multiple tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
