VCR-Bench：面向视频思维链推理的综合评估框架
VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning
AI总结：
本文提出VCR-Bench基准，通过859个视频和1,034个逐步标注感知与推理步骤的问答对，系统评估大视觉语言模型的视频思维链推理能力，并揭示其在时空信息处理上的关键瓶颈。
摘要：
思维链（CoT）推理的进步显著增强了大语言模型（LLMs）和大视觉语言模型（LVLMs）的能力。然而，针对视频CoT推理的严格评估框架仍然缺失。当前的视频基准无法充分评估推理过程，也无法揭示失败究竟源于感知能力还是推理能力的不足。因此，我们提出了VCR-Bench，这是一个旨在全面评估LVLMs视频思维链推理能力的新型基准。VCR-Bench包含859个视频，涵盖多种视频内容和时长，以及1,034个高质量问答对。每个问答对都经过人工标注，配有逐步的CoT推理依据，其中每一步都被标记以表明其与感知能力或推理能力的关联。此外，我们设计了七个不同的任务维度，并提出CoT分数，以基于逐步标记的CoT推理依据来评估整个CoT过程。在VCR-Bench上的大量实验揭示了当前LVLMs的显著局限性。即使是表现最好的模型o1，也仅达到62.8%的CoT分数和56.7%的准确率，而大多数模型得分低于40%。实验表明，大多数模型在感知步骤上的得分低于推理步骤，揭示了LVLMs在复杂视频推理中时空信息处理方面的关键瓶颈。CoT分数与准确率之间稳健的正相关关系证实了我们评估框架的有效性，并凸显了CoT推理在解决复杂视频推理任务中的关键作用。我们希望VCR-Bench能够作为一个标准化的评估框架，并揭示复杂视频推理任务中的实际缺陷。
英文摘要：
The advancement of Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs) and large vision-language models (LVLMs). However, a rigorous evaluation framework for video CoT reasoning remains absent. Current video benchmarks fail to adequately assess the reasoning process and expose whether failures stem from deficiencies in perception or reasoning capabilities. Therefore, we introduce VCR-Bench, a novel benchmark designed to comprehensively evaluate LVLMs' Video Chain-of-Thought Reasoning capabilities. VCR-Bench comprises 859 videos spanning a variety of video content and durations, along with 1,034 high-quality question-answer pairs. Each pair is manually annotated with a stepwise CoT rationale, where every step is tagged to indicate its association with the perception or reasoning capabilities. Furthermore, we design seven distinct task dimensions and propose the CoT score to assess the entire CoT process based on the stepwise tagged CoT rationals. Extensive experiments on VCR-Bench highlight substantial limitations in current LVLMs. Even the top-performing model, o1, only achieves a 62.8% CoT score and an 56.7% accuracy, while most models score below 40%. Experiments show most models score lower on perception than reasoning steps, revealing LVLMs' key bottleneck in temporal-spatial information processing for complex video reasoning. A robust positive correlation between the CoT score and accuracy confirms the validity of our evaluation framework and underscores the critical role of CoT reasoning in solving complex video reasoning tasks. We hope VCR-Bench to serve as a standardized evaluation framework and expose the actual drawbacks in complex video reasoning task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
