当思考带来痛苦：通过帧重复缓解视频推理中的视觉遗忘
When Thinking Hurts: Mitigating Visual Forgetting in Video Reasoning via Frame Repetition
AI总结：
本文提出FrameRepeat框架，通过轻量级重复评分模块和Add-One-In策略，有效增强视频推理中的视觉线索，提升模型性能和泛化能力。
摘要：
近年来，多模态大语言模型（MLLMs）通过整合链式推理（CoT）在复杂视觉任务中展现出巨大潜力。然而，在视频问答中，扩展的推理过程并不总能带来性能提升，甚至可能因「视觉锚点漂移」导致退化，其中模型越来越依赖自动生成的文本，忽视视觉输入并导致幻觉。现有缓解方法通常引入特定机制让模型在推理时重新关注视觉输入，但这些方法往往带来高昂的训练成本，并在不同架构间泛化性差。为此，我们提出FrameRepeat，一个自动增强框架，其包含轻量级重复评分模块，使视频大语言模型能够自主识别应加强的帧。我们引入了一种新的训练策略Add-One-In（AOI），利用MLLM输出概率生成监督信号表示重复增益。这可用于训练一个帧评分网络，引导帧重复行为。在多个模型和数据集上的实验结果表明，FrameRepeat在推理过程中增强重要视觉线索方面既有效又具有泛化能力。
英文摘要：
Recently, Multimodal Large Language Models (MLLMs) have demonstrated significant potential in complex visual tasks through the integration of Chain-of-Thought (CoT) reasoning. However, in Video Question Answering, extended thinking processes do not consistently yield performance gains and may even lead to degradation due to ``visual anchor drifting'', where models increasingly rely on self-generated text, sidelining visual inputs and causing hallucinations. While existing mitigations typically introduce specific mechanisms for the model to re-attend to visual inputs during inference, these approaches often incur prohibitive training costs and suffer from poor generalizability across different architectures. To address this, we propose FrameRepeat, an automated enhancement framework which features a lightweight repeat scoring module that enables Video-LLMs to autonomously identify which frames should be reinforced. We introduce a novel training strategy, Add-One-In (AOI), that uses MLLM output probabilities to generate supervision signals representing repeat gain. This can be used to train a frame scoring network, which guides the frame repetition behavior. Experimental results across multiple models and datasets demonstrate that FrameRepeat is both effective and generalizable in strengthening important visual cues during the reasoning process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
