TriCLIP-3D：一种基于CLIP的三模态3D视觉定位统一参数高效框架
TriCLIP-3D: A Unified Parameter-Efficient Framework for Tri-Modal 3D Visual Grounding based on CLIP
AI总结：
针对现有3D视觉定位方法模型复杂、训练效率低的问题，提出基于CLIP的三模态统一参数高效框架TriCLIP-3D，通过适配器微调、GARF模块等实现端到端定位，参数量大减且性能显著提升。
摘要：
3D视觉定位使具身智能体能够根据人类指令理解真实3D环境中的视觉信息，这对具身智能至关重要。现有3D视觉定位方法通常依赖独立编码器处理不同模态（如RGB图像、文本和3D点云），导致模型庞大复杂，训练效率低下。虽然部分方法将CLIP等预训练2D多模态模型用于3D任务，但它们仍难以将点云数据与2D编码器对齐，因此这些方法依旧依赖3D编码器进行特征提取，进一步增加了模型复杂度和训练低效性。本文提出一种统一的2D预训练多模态网络来处理全部三种模态（RGB图像、文本和点云），显著简化了架构。该框架借助采用基于适配器微调的2D CLIP双模态模型，有效适配三模态场景，提升了跨模态的适配性与性能。我们设计的几何感知2D-3D特征恢复与融合（Geometric-Aware 2D-3D Feature Recovery and Fusion, GARF）模块，用于融合来自点云和图像的几何多尺度特征。随后我们整合文本特征完成最终的模态融合，并引入多模态解码器以促进深度跨模态理解。综上，我们的方法实现了三种模态的统一特征提取与融合，构建出端到端的3D视觉定位模型。与基线方法相比，我们的方法可训练参数量减少约58%，同时在3D检测任务上提升6.52%，在3D视觉定位任务上提升6.25%。
英文摘要：
3D visual grounding allows an embodied agent to understand visual information in real-world 3D environments based on human instructions, which is crucial for embodied intelligence. Existing 3D visual grounding methods typically rely on separate encoders for different modalities (e.g., RGB images, text, and 3D point clouds), resulting in large and complex models that are inefficient to train. While some approaches use pre-trained 2D multi-modal models like CLIP for 3D tasks, they still struggle with aligning point cloud data to 2D encoders. As a result, these methods continue to depend on 3D encoders for feature extraction, further increasing model complexity and training inefficiency. In this paper, we propose a unified 2D pre-trained multi-modal network to process all three modalities (RGB images, text, and point clouds), significantly simplifying the architecture. By leveraging a 2D CLIP bi-modal model with adapter-based fine-tuning, this framework effectively adapts to the tri-modal setting, improving both adaptability and performance across modalities. Our Geometric-Aware 2D-3D Feature Recovery and Fusion (GARF) module is designed to fuse geometric multi-scale features from point clouds and images. We then integrate textual features for final modality fusion and introduce a multi-modal decoder to facilitate deep cross-modal understanding. Together, our method achieves unified feature extraction and fusion across the three modalities, enabling an end-to-end 3D visual grounding model. Compared to the baseline, our method reduces the number of trainable parameters by approximately 58\%, while achieving a 6.52\% improvement in the 3D detection task and a 6.25\% improvement in the 3D visual grounding task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
