arXiv 2506.099352026-03-23cs.CV

LEO-VL：面向可扩展3D视觉-语言学习的高效场景表示

LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning

Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）

Jiangyong Huang, Xiaojian Ma, Xiongkun Linghu, Junchao He, Qing Li, Song-Chun Zhu, Yixin Chen, Baoxiong Jia, Siyuan Huang

备注：: Project page: https://leo-vl.github.io

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LEO-VL，一种基于高效场景表示CFG的3D视觉-语言模型，通过改进训练数据和引入SceneDPO提升鲁棒性，在多个3D-VL基准测试中取得最佳性能。

摘要：

开发能够理解3D场景的视觉-语言模型（VLMs）一直是长期研究目标。尽管近期进展显著，3D VLMs在空间推理和鲁棒性方面仍面临挑战。本文识别出三个阻碍其发展的关键障碍：（1）场景表示受限于容量-效率权衡，阻碍可扩展学习；（2）训练数据缺乏全面方案，任务和场景领域多样性有限；（3）模型存在鲁棒性不足且缺乏有效后训练。为解决这些挑战，我们首先提出压缩特征网格（CFG），一种高效的场景表示，显著减少令牌开销同时保持强感知能力。基于CFG，我们引入LEO-VL，一种在超过70万3D视觉-语言（3D-VL）数据上训练的3D VLM，涵盖四个现实室内领域和五种任务如描述和对话。为进一步提升鲁棒性，我们提出SceneDPO，一种新的后训练目标，结合答案和场景间的对比信号。LEO-VL在多个3D-VL基准测试中取得最佳性能，如SQA3D、Beacon3D和Scan2Cap。广泛分析强调了CFG的效率，并提供了关键见解，如任务和场景多样性的重要性、数据质量对有效扩展的优先性，以及SceneDPO的优势。

英文摘要：

Developing vision-language models (VLMs) capable of understanding 3D scenes has been a longstanding research goal. Despite recent progress, 3D VLMs still struggle with spatial reasoning and robustness. We identify three key obstacles hindering their progress: (1) scene representation is constrained by a capacity-efficiency trade-off, which impedes scalable learning; (2) training data lacks a comprehensive scheme, with limited diversity across tasks and scene domains; and (3) models exhibit robustness deficiencies and lack effective post-training. To address these challenges, we first propose condensed feature grid (CFG), an efficient scene representation that significantly reduces token overhead while preserving strong perceptual capacity. Building on CFG, we introduce LEO-VL, a 3D VLM trained on over 700k 3D vision-language (3D-VL) data spanning four real-world indoor domains and five tasks such as captioning and dialogue. To further improve robustness, we propose SceneDPO, a novel post-training objective that incorporates contrastive signals across both answers and scenes. LEO-VL achieves state-of-the-art performance on various 3D-VL benchmarks, such as SQA3D, Beacon3D, and Scan2Cap. Extensive analyses highlight the efficiency of CFG and provide key insights such as the importance of task and scene diversity, the priority of data quality for effective scaling, and the advantages of SceneDPO.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。