面向统一3D表示学习的多模态关系蒸馏
Multi-modal Relation Distillation for Unified 3D Representation Learning
AI总结：
针对现有3D点云多模态预训练忽略样本结构关系的问题，提出多模态关系蒸馏框架MRD，蒸馏大型VLM到3D骨干，捕捉模态内与跨模态关系，在零样本分类、跨模态检索任务上取得SOTA性能。
摘要：
3D点云多模态预训练领域的最新进展通过对齐3D形状与其对应2D图像、语言描述之间的异构特征，已展现出可观成效。但当前的直接方案往往忽略样本间复杂的结构关系，可能限制多模态学习的全部潜力。为解决该问题，我们提出多模态关系蒸馏（Multi-modal Relation Distillation, MRD）这一三模态预训练框架，旨在将成熟的大型视觉语言模型（Vision-Language Models, VLM）有效蒸馏到3D骨干网络中。MRD同时捕捉各模态内部的内关系与不同模态间的跨关系，生成更具判别性的3D形状表示。值得注意的是，MRD在下游零样本分类任务与跨模态检索任务上均取得显著提升，刷新了当前最优性能。
英文摘要：
Recent advancements in multi-modal pre-training for 3D point clouds have demonstrated promising results by aligning heterogeneous features across 3D shapes and their corresponding 2D images and language descriptions. However, current straightforward solutions often overlook intricate structural relations among samples, potentially limiting the full capabilities of multi-modal learning. To address this issue, we introduce Multi-modal Relation Distillation (MRD), a tri-modal pre-training framework, which is designed to effectively distill reputable large Vision-Language Models (VLM) into 3D backbones. MRD aims to capture both intra-relations within each modality as well as cross-relations between different modalities and produce more discriminative 3D shape representations. Notably, MRD achieves significant improvements in downstream zero-shot classification tasks and cross-modality retrieval tasks, delivering new state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
