ACQUIRED：一个用于回答现实生活视频中反事实问题的数据集
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos
AI总结：
本文构建包含3.9K真实视频、覆盖多事件类型与物理、社会、时间三维反事实推理的 ACQUIRED 数据集，实验显示先进模型与人类差距超过13%。
摘要：
多模态反事实推理对 AI 系统而言是一项至关重要却颇具挑战性的能力。它涉及基于视觉和语言输入预测假设情形下的结果，使 AI 模型能够从失败中学习并探索假设场景。尽管其重要性显著，但目前针对多模态模型反事实推理能力的数据集寥寥无几。在这些现有数据集中，它们只涵盖合成环境或特定类型事件（例如交通事故）上的推理，因此难以在多样化的真实世界场景和推理维度中可靠地基准测试模型的泛化能力。为克服这些局限，我们开发了一个视频问答数据集 ACQUIRED：它包含 3.9K 个带标注视频，涵盖广泛的事件类型，并纳入第一人称和第三人称视角，从而确保聚焦真实世界的多样性。此外，每个视频都配有跨越三个不同推理维度的问题，包括物理、社会和时间维度，能够从多个方面全面评估模型的反事实能力。我们使用若干最先进的纯语言模型和多模态模型在该数据集上进行基准测试，实验结果表明，模型与人类之间存在显著的性能差距（>13%）。这些发现表明，多模态反事实推理仍是一个开放性挑战，而 ACQUIRED 是一个全面且可靠的基准，可启发该方向的未来研究。
英文摘要：
Multimodal counterfactual reasoning is a vital yet challenging ability for AI systems. It involves predicting the outcomes of hypothetical circumstances based on vision and language inputs, which enables AI models to learn from failures and explore hypothetical scenarios. Despite its importance, there are only a few datasets targeting the counterfactual reasoning abilities of multimodal models. Among them, they only cover reasoning over synthetic environments or specific types of events (e.g. traffic collisions), making them hard to reliably benchmark the model generalization ability in diverse real-world scenarios and reasoning dimensions. To overcome these limitations, we develop a video question answering dataset, ACQUIRED: it consists of 3.9K annotated videos, encompassing a wide range of event types and incorporating both first and third-person viewpoints, which ensures a focus on real-world diversity. In addition, each video is annotated with questions that span three distinct dimensions of reasoning, including physical, social, and temporal, which can comprehensively evaluate the model counterfactual abilities along multiple aspects. We benchmark our dataset against several state-of-the-art language-only and multimodal models and experimental results demonstrate a significant performance gap (>13%) between models and humans. The findings suggest that multimodal counterfactual reasoning remains an open challenge and ACQUIRED is a comprehensive and reliable benchmark for inspiring future research in this direction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
