Minerva-Ego：眼动视频理解的空间时间提示
Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
AI总结：
本文提出Minerva-Ego基准，通过多步骤多模态问题和密集标注的时空推理轨迹评估眼动视频理解模型，发现提示'何时何地'显著提升性能。
摘要：
视频推理模型是眼动和具身智能体的核心组成部分。然而，标准评估模型的基准仅提供输出评估（例如回答问题），而不评估中间推理步骤，且大多数仅提供文本领域的答案。我们引入了Minerva-Ego，一个用于评估复杂眼动视觉推理的基准。我们扩展了最近高质量的视频数据源，这些数据源来自眼动/具身设置，并添加了一组具有挑战性的多步骤多模态问题和密集标注的时空推理轨迹。基准测试实验表明，最先进的模型与人类表现之间仍有较大差距。为了深入研究这一差距，我们对数据集中的每个推理轨迹标注了所需解决问题的对象，作为时空掩码标注。通过广泛的评估，我们发现提示前沿模型以'哪里'和'何时'的提示来查看，能显著提高性能。Minerva-Ego可在https://github.com/google-deepmind/neptune下载。
英文摘要：
Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
