DreamFrame：通过自动生成的问答与风格一致的关键帧增强视频理解
DreamFrame: Enhancing Video Understanding via Automatically Generated QA and Style-Consistent Keyframes
AI总结：
为解决视频理解LVLMs下游适配的标注难题，提出三阶段框架DreamFrame，通过LLM生成情节、QA对及风格一致的关键帧构建数据集，微调的DreamFrame-7B模型在多项基准上显著超越同规模模型。
摘要：
近期用于视频理解的大型视觉语言模型（LVLMs）主要通过从在线平台抓取的各种视频进行微调。现有数据集（如ActivityNet）在有效用于调整LVLMs之前，需要大量人力进行结构化和标注。虽然当前的LVLMs主要在广泛、通用的设置下基于现有数据集进行训练，但将其适应到特定的下游场景仍然具有挑战性，因为收集和标注特定任务的视频是高度劳动密集且耗时的。为了解决这个问题，我们提出了一个名为DreamFrame的三阶段框架，用于自动生成风格一致的关键帧和相应的问答（QA）对，以支持LVLM指令微调。DreamFrame以类似电影的方式生成数据集。首先，我们利用LLM生成结构化的电影情节，包括电影先验信息（如概述和风格）、帧描述和与情节相关的QA对，并采用故事扩展策略来缓解上下文长度限制。然后，为了确保生成帧之间的视觉一致性，我们设计了一个风格固化过程，通过嵌入学习策略保持一致的风格。最后，将帧描述和风格嵌入整合以生成连贯的关键帧。使用DreamFrame，我们构建了一个包含约1k个风格化类关键帧视频和100k个多样化QA对的数据集。在各种LVLM架构上的广泛微调实验证明了所提出数据集的有效性。此外，基于所提出的数据集，我们微调了一个名为DreamFrame-7B的新LVLM，它在不同的基准测试中显著超越了先前相似规模的LVLMs。
英文摘要：
Recent large vision-language models (LVLMs) for video understanding are primarily fine-tuned with various videos scraped from online platforms. Existing datasets, such as ActivityNet, require considerable human labor for structuring and annotation before effectively utilized for tuning LVLMs. While current LVLMs are primarily trained on existing datasets in broad, general-purpose settings, adapting them to specific downstream scenarios remains challenging, as collecting and annotating task-specific videos is highly labor-intensive and time-consuming. To address this issue, we propose a three-stage framework named DreamFrame for automatically generating style-consistent keyframes and corresponding question-answer (QA) pairs to support LVLM instruction tuning. DreamFrame generates datasets in a movie-like manner. First, we utilize an LLM to generate structured movie plots including movie prior information (like overview and style), frame descriptions and plot-related QA pairs, with a story expansion strategy to mitigate context length limitations.Then, to ensure visual consistency across generated frames, we design a Style Immobilization Process which maintains consistent style through an embedding learning strategy. Finally, frame descriptions and style embeddings are integrated to produce coherent keyframes. Using DreamFrame, we construct a dataset comprising approximately 1k stylized keyframe-like videos and 100k diverse QA pairs. Extensive fine-tuned experiments on various LVLM architectures demonstrate the effectiveness of the proposed dataset. Furthermore, based on the proposed dataset, we fine-tune a new LVLM named DreamFrame-7B, which significantly surpasses the previous similar-sized LVLMs across different benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
