arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.00547cs.CVcs.AI
收藏

EgoGapBench：多智能体场景中自我中心动作选择的基准测试

EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes

KAIST AI（韩国科学技术院人工智能研究所）；Sogang University（成均馆大学）；Ministry of Science and ICT（科学技术信息通信部）

Jihyeok Jung, Jeewu Lee, Sanghyeop Kim, Chanhee Han, Seong Joon Oh

备注：
15 pages, 2 figures, 8 tables. Code and benchmark are available at https://github.com/jhCOR/EgoGapBench
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出EgoGapBench基准，用于评估多智能体场景中的自我中心动作选择能力，发现多模态大语言模型表现远逊于人类，且现有自我中心数据微调无法弥合差距。

摘要：

现有的自我中心基准主要通过第一人称视角数据构建自我中心设置，这使得难以单独评估自我中心视角本身。然而，理解第一人称输入和采取自我中心视角是可分离的能力，特别是在缺乏第一人称身体线索或存在其他智能体时。为了隔离自我中心视角理解，我们引入了EgoGapBench，一个用于测量多智能体自我中心场景中动作选择的诊断基准。我们将此基准测量的能力定义为自我中心动作选择（EAS）：在其他智能体存在的情况下，从智能体自身视角选择合适动作。在EgoGapBench上，人类回答可靠，而开源和闭源的多模态大语言模型（MLLM）表现显著较差，并系统性地选择其他可见智能体执行的动作。在现有自我中心数据上微调无法弥合这一差距，甚至可能有害。相反，在EgoGapBench训练数据上微调提高了准确性，但未达到人类水平。这些结果表明，EAS难以仅从第一人称视角数据中获取，并且MLLM不仅应针对场景理解进行评估和训练，还应针对自我中心动作选择进行评估和训练。

英文摘要：

Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when first-person body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent's perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑