STEP：通过时空图引导的自训练增强视频大语言模型的组合推理能力
STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training
AI总结：
本文提出STEP方法，利用时空场景图引导视频大语言模型自生成推理数据，通过思维链监督提升多步组合推理能力，实验显示在复杂推理任务上提升21.3%。
摘要：
视频大语言模型（Video-LLMs）最近在基础视频理解任务（如字幕生成和粗粒度问答）中表现出强大的性能，但在需要跨对象关系、交互和事件进行多步时空推理的组合推理方面仍存在困难。增强这一能力的障碍包括大量的人工劳动、现有数据缺乏时空组合性以及缺乏明确的推理监督。在本文中，我们提出了STEP，一种新颖的图引导自训练方法，使视频大语言模型能够从任意原始视频中生成富含推理的微调数据以自我改进。具体来说，我们首先诱导多样视频的时空场景图（STSG）表示，以捕获细粒度、多粒度的视频语义。然后，STSG指导推导具有思维链（CoT）理由的多步推理问答（QA）数据。答案和理由都被整合为训练目标，旨在通过对显式推理步骤的监督来增强模型的推理能力。实验结果表明，STEP在不同规模的模型中均有效，在需要三个或更多推理步骤的任务中显著提高了21.3%。此外，它在组合推理和综合理解基准上以最少的自生成理由丰富的训练样本实现了优越的性能，突显了其广泛的适用性和巨大潜力。
英文摘要：
Video Large Language Models (Video-LLMs) have recently shown strong performance in basic video understanding tasks, such as captioning and coarse-grained question answering, but struggle with compositional reasoning that requires multi-step spatio-temporal inference across object relations, interactions, and events. The hurdles to enhancing this capability include extensive manual labor, the lack of spatio-temporal compositionality in existing data and the absence of explicit reasoning supervision. In this paper, we propose STEP, a novel graph-guided self-training method that enables Video-LLMs to generate reasoning-rich fine-tuning data from any raw videos to improve itself. Specifically, we first induce Spatio-Temporal Scene Graph (STSG) representation of diverse videos to capture fine-grained, multi-granular video semantics. Then, the STSGs guide the derivation of multi-step reasoning Question-Answer (QA) data with Chain-of-Thought (CoT) rationales. Both answers and rationales are integrated as training objective, aiming to enhance model's reasoning abilities by supervision over explicit reasoning steps. Experimental results demonstrate the effectiveness of STEP across models of varying scales, with a significant 21.3\% improvement in tasks requiring three or more reasoning steps. Furthermore, it achieves superior performance with a minimal amount of self-generated rationale-enriched training samples in both compositional reasoning and comprehensive understanding benchmarks, highlighting the broad applicability and vast potential.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
