arXiv 2505.041472025-05-07cs.CVcs.AI

R^3-VQA：通过视频社会推理“读懂房间”

R^3-VQA: "Read the Room" by Video Social Reasoning

School of Intelligence Science and Technology, Peking University（北京理工大学智能科学与技术学院）；College of Artificial Intelligence, Xi’an Jiaotong University（西安交通大学人工智能学院）；State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence（一般人工智能国家重点实验室）；Yuanpei College, Peking University（北京大学元培学院）；Tsinghua University（清华大学）；University of California, Los Angeles（加州大学洛杉矶分校）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）

Lixing Niu, Jiapeng Li, Xingping Yu, Shu Wang, Ruining Feng, Bo Wu, Ping Wei, Yisen Wang, Lifeng Fan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出视频社会推理数据集与基准 R^3-VQA，涵盖社会事件理解、心理状态估计和社会因果推理，并系统评估现有大型视觉语言模型，发现其与人类水平仍有显著差距，同时验证心智理论提示的有效性。

摘要：

“读懂房间”是人类日常生活中的一项重要社会推理能力。人类能够从细微的社会线索中推断他人的心理状态。以往的社会推理任务和数据集缺乏复杂性（例如场景简单、交互基础、心理状态变量不完整、单步推理等），远未达到现实社会互动中所呈现的挑战水平。在本文中，我们贡献了一个有价值、高质量且全面的视频数据集，名为 R^3-VQA，该数据集在复杂社会场景中对社会事件和心理状态（即信念、意图、欲望和情绪）以及相应的社会因果链进行了精确且细粒度的标注。此外，我们还包含了人工标注和模型生成的问答对。我们的任务 R^3-VQA 包括三个方面：社会事件理解、心理状态估计和社会因果推理。作为一个基准，我们全面评估了当前最先进的大型视觉语言模型（LVLMs）的社会推理能力和一致性。综合实验表明：(i) 在复杂社会场景中，LVLMs 仍远未达到人类水平的一致社会推理；(ii) 心智理论（ToM）提示可以帮助 LVLMs 在社会推理任务上表现更好。我们在补充材料中提供了部分数据集和代码，并将在录用后发布完整的数据集和代码。

英文摘要：

"Read the room" is a significant social reasoning capability in human daily life. Humans can infer others' mental states from subtle social cues. Previous social reasoning tasks and datasets lack complexity (e.g., simple scenes, basic interactions, incomplete mental state variables, single-step reasoning, etc.) and fall far short of the challenges present in real-life social interactions. In this paper, we contribute a valuable, high-quality, and comprehensive video dataset named R^3-VQA with precise and fine-grained annotations of social events and mental states (i.e., belief, intent, desire, and emotion) as well as corresponding social causal chains in complex social scenarios. Moreover, we include human-annotated and model-generated QAs. Our task R^3-VQA includes three aspects: Social Event Understanding, Mental State Estimation, and Social Causal Reasoning. As a benchmark, we comprehensively evaluate the social reasoning capabilities and consistencies of current state-of-the-art large vision-language models (LVLMs). Comprehensive experiments show that (i) LVLMs are still far from human-level consistent social reasoning in complex social scenarios; (ii) Theory of Mind (ToM) prompting can help LVLMs perform better on social reasoning tasks. We provide some of our dataset and codes in supplementary material and will release our full dataset and codes upon acceptance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。