通过自我场景增强在多模态大语言模型中强化自我中心空间感知
Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation
浏览论文内容
中文总结 AI 辅助
研究如何强化多模态大语言模型的自我中心空间感知，提出自我场景增强框架ESA，利用自我元素图作为中间表示，通过视觉基础模型增强空间感知，在EgoTextVQA基准上取得显著性能提升。
中文摘要 AI 辅助
自我中心视觉问答（VQA）作为使多模态大语言模型（MLLMs）与现实世界交互的重要任务受到广泛关注。然而，现有MLLMs由于空间感知能力有限，在复杂的自我中心场景中难以进行有效的空间推理。为此，我们引入了自我场景增强（ESA），这是一个自我中心空间感知框架，由提出的自我元素图驱动，从自我中心视角积极增强空间感知能力。我们的核心见解是利用自我元素图作为中间表示，通过视觉基础模型增强MLLMs的自我中心空间感知。具体来说，我们1）构建自我元素图，封装并整合视觉基础模型启用的自我中心空间特征；2）通过自我元素图增强MLLMs对自我视角场景的空间感知能力。我们提出的ESA框架在EgoTextVQA基准上有显著的性能提升。在室内设置中获得了8.14%的增益，在室外设置中获得了8.72%的增益。此外，我们的ESA在室内设置的购物子集中表现出最显著的性能提升。项目代码已公开。
英文摘要
Egocentric Visual Question Answering (VQA) has attracted widespread attention as an important task for enabling Multimodal Large Language Models (MLLMs) to interact with the real world. However, existing MLLMs struggle to perform effective spatial reasoning in complex egocentric scenes due to their limited spatial perception capabilities. To this end, we introduce Ego Scene Augmentation (ESA), an egocentric spatial perception framework, which actively enhances the spatial perception capabilities from the egocentric perspective, powered by the proposed Ego-element Graph. Our core insight is leveraging the Ego-element Graph as an intermediary representation to augment the egocentric spatial perception of MLLMs via visual foundational models. Specifically, we 1) construct the Ego-element Graph, which encapsulates and integrates egocentric spatial features enabled by visual foundational models; 2) enhance the spatial perception capabilities of MLLMs via the Ego-element Graph for ego-perspective scenes. Our proposed ESA framework presents significant performance improvement on the EgoTextVQA benchmark. We achieve an 8.14% gain on the indoor setting and an 8.72% gain on the outdoor setting. Furthermore, our ESA shows the most impressive performance improvement in the shopping subset of the indoor setting. The project code is publicly available.
发表机构
- The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
- Guangxi Zhuang Autonomous Region Information Center（广西壮族自治区信息中心）
- The Hong Kong University of Science and Technology（香港科技大学）
机构由 AI 辅助整理，请以论文原文为准。
