arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.22862cs.CVcs.LG
收藏

看得见的链,答不出的答案:针对视频MME上强制思维链的多方面评估方案

Chains That See, Answers That Don't: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MME

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Northeastern University(东北大学);Southern Methodist University(南卫理公会大学)

Zhichao Fan, Yanhang Li, Zexin Zhuang

AI总结:

提出三探针评估方案检验强制思维链在视频问答中的可靠性,应用于Qwen2.5-VL发现思维链强依赖视频但未提升准确率,甚至在小模型上导致下降。

摘要:

强制思维链(CoT)被广泛认为能使视觉语言模型在视频问答中更可靠。我们提出一个包含三个探针的小型评估方案来检验这一假设:直接、CoT、答案优先和无视频条件下的配对准确率;对CoT链的反事实视频交换诊断;以及四级视觉退化阶梯。每个探针在严格和宽松的正则表达式评分器下报告,并对稿件声明的主要族进行多重性校正。将该方案应用于Qwen2.5-VL在Video-MME子集上,得到两部分发现。CoT链强烈依赖于视频:交换输入视频会破坏链的重叠并翻转大多数最终字母,这与“模板链”零假设的预测相反。然而,在相同数据上,强制CoT并未提高多项选择题准确率,并且在较小的7B模型上,在后验主要评分器选择下,它产生了微小但统计上显著的下降。我们不声称这泛化到Qwen2.5-VL / Video-MME实例之外;原始响应和单个重计算脚本将与补充材料一起发布,以便每个数字都可以重新推导。

英文摘要:

Forced chain-of-thought (CoT) is widely assumed to make vision-language models more reliable on video question answering. We propose a small three-probe evaluation recipe to test that assumption: paired accuracy across direct, CoT, answer-first, and no-video conditions; a counterfactual video-swap diagnostic over the CoT chains; and a four-rung visual-degradation ladder. Each probe is reported under both a strict and a permissive regex scorer, with multiplicity correction over a manuscript-declared primary family. Applied to Qwen2.5-VL on Video-MME subsets, the recipe returns a two-part finding. The CoT chains are strongly video-conditioned: swapping the input video collapses chain overlap and flips most final letters, the opposite of what a "boilerplate-chain" null would predict. Yet on the same data, forced CoT does not improve MCQ accuracy, and on the smaller 7B model it produces a small but statistically supported drop under a post-hoc primary scorer choice. We do not claim this generalizes beyond the Qwen2.5-VL / Video-MME instantiation; the raw responses and a single recomputation script will be released with the supplementary material so every number can be re-derived.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑