利用电影中的叙事套路研究大语言模型的视频推理能力
Investigating Video Reasoning Capability of Large Language Models with Tropes in Movies
AI总结：
本文提出电影叙事套路数据集TiM，用于评估大语言模型的抽象感知与长程组合推理能力，并通过FEVoRI和ConQueR增强视觉编程，显著提升性能但仍低于人类水平。
摘要：
大语言模型（LLMs）不仅在语言任务中表现出有效性，在视频推理中也同样如此。本文引入了一个新的数据集——电影叙事套路（Tropes in Movies, TiM），旨在作为测试平台，用于探索两项关键但此前被忽视的视频推理技能：（1）抽象感知：理解并标记视频中的抽象概念；（2）长程组合推理：规划并整合中间推理步骤，以理解包含大量帧的长视频。TiM 利用电影叙事中的套路，评估了当前最先进的基于 LLM 的方法的推理能力。我们的实验表明，当前的方法——包括 Captioner-Reasoner、大型多模态模型指令微调（Large Multimodal Model Instruction Fine-tuning）和视觉编程（Visual Programming）——在面对抽象感知和长程组合推理的挑战时，仅略优于随机基线。为解决这些不足，我们提出了 Face-Enhanced Viper of Role Interactions（FEVoRI）和 Context Query Reduction（ConQueR），它们通过促进角色交互感知，并在推理过程中逐步细化电影上下文和套路查询，来增强视觉编程，从而显著将性能提升了 15 个 F1 点。然而，这一性能仍落后于人类水平（40 对 65 F1）。此外，我们引入了一种新的协议，通过使用抽象语法树（AST）分析视觉编程生成的代码，来评估抽象感知和长程组合推理对任务解决的必要性，从而证实了 TiM 复杂度的增加。数据集和代码可在以下地址获取：https://ander1119.github.io/TiM
英文摘要：
Large Language Models (LLMs) have demonstrated effectiveness not only in language tasks but also in video reasoning. This paper introduces a novel dataset, Tropes in Movies (TiM), designed as a testbed for exploring two critical yet previously overlooked video reasoning skills: (1) Abstract Perception: understanding and tokenizing abstract concepts in videos, and (2) Long-range Compositional Reasoning: planning and integrating intermediate reasoning steps for understanding long-range videos with numerous frames. Utilizing tropes from movie storytelling, TiM evaluates the reasoning capabilities of state-of-the-art LLM-based approaches. Our experiments show that current methods, including Captioner-Reasoner, Large Multimodal Model Instruction Fine-tuning, and Visual Programming, only marginally outperform a random baseline when tackling the challenges of Abstract Perception and Long-range Compositional Reasoning. To address these deficiencies, we propose Face-Enhanced Viper of Role Interactions (FEVoRI) and Context Query Reduction (ConQueR), which enhance Visual Programming by fostering role interaction awareness and progressively refining movie contexts and trope queries during reasoning processes, significantly improving performance by 15 F1 points. However, this performance still lags behind human levels (40 vs. 65 F1). Additionally, we introduce a new protocol to evaluate the necessity of Abstract Perception and Long-range Compositional Reasoning for task resolution. This is done by analyzing the code generated through Visual Programming using an Abstract Syntax Tree (AST), thereby confirming the increased complexity of TiM. The dataset and code are available at: https://ander1119.github.io/TiM
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
