EgoExo-Con: 探索视角不变视频时间理解
EgoExo-Con: Exploring View-Invariant Video Temporal Understanding
AI总结：
提出EgoExo-Con基准测试，评估视频大模型在不同视角下的时间理解一致性，并设计View-GRPO强化学习框架提升跨视角一致性。
摘要：
视频大模型在从不同视角捕捉同一事件时，是否具有一致的时间理解？为研究这一问题，我们引入了EgoExo-Con(sistency)，一个包含同步第一人称和第三人称视频对的基准测试，并配有经过人工精炼的查询，确保所有概念在两个视角中都可见。EgoExo-Con强调两个时间理解任务：时间验证和时间定位。它不仅评估正确性，还评估跨视角的一致性。我们的分析揭示了现有视频大模型的两个关键局限性：(1) 模型通常无法保持一致性，其结果远差于单视角性能。(2) 当使用两个视角的同步视频进行简单微调时，模型的一致性有所提升，但通常不如在单一视角上训练的模型。为改进，我们提出了View-GRPO，一种新颖的强化学习框架，有效增强视角特定的时间推理，同时鼓励跨视角的一致理解。我们的方法展示了其卓越的时间理解能力，特别是在提升跨视角一致性方面。所有资源已在此https URL公开。
英文摘要：
Do Video-LLMs have consistent temporal understanding when videos capture the same event from different viewpoints? To study this question, we introduce EgoExo-Con(sistency), a benchmark of synchronized egocentric and exocentric video pairs with human-refined queries that ensure all concepts are visible in both viewpoints. EgoExo-Con emphasizes two temporal understanding tasks: Temporal Verification and Temporal Grounding. It evaluates not only correctness but consistency across viewpoints. Our analysis reveals two critical limitations of existing Video-LLMs: (1) models often fail to maintain consistency, with results far worse than their single-view performances. (2) When naively finetuned with synchronized videos of both viewpoints, the models show improved consistency but often underperform those trained on a single view. For improvements, we propose View-GRPO, a novel reinforcement learning framework that effectively strengthens view-specific temporal reasoning while encouraging consistent comprehension across viewpoints. Our method demonstrates its superior temporal understanding capabilities, especially for improving cross-view consistency. All resources have been made available at https://minjoong507.github.io/projects/EgoExo-Con/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
