Sparrow：基于文本到图像增强的数据高效视频大语言模型
Sparrow: Data-Efficient Video-LLM with Text-to-Image Augmentation
AI总结：
Sparrow 针对视频大语言模型数据规模扩展中指令多样性不足导致学习效率低的问题，提出从纯文本指令合成视频样本来增强训练，在更少样本下实现与大规模训练相当甚至更优的性能，并提升长视频理解能力。
摘要：
近年来，多模态大语言模型（MLLMs）在视觉理解领域取得了成功。这些模型的成功在很大程度上可归因于主导性的规模法则，即更大的参数规模和数据量有助于获得更好的性能。值得注意的是，数据规模扩展主要由自动数据流水线驱动，这些流水线侧重于大语言模型的自指令生成。这一范式在相当长的时间内被视为理所当然，但对使用这些数据进行规模扩展的有效性研究却长期被忽视。在此背景下，本工作重新审视了使用合成数据进行规模扩展的问题，并从以数据为中心的视角专注于开发视频大语言模型。我们的主要研究方法包括使用视频数据微调预训练的图像大语言模型，并通过数据规模扩展来考察学习效率。初步实验结果表明，简单地扩大视频数据样本规模会出现学习效率低下的现象，通过我们的探查，这一现象可归因于指令多样性的缺乏。针对这一问题，我们提出了一种名为 Sparrow 的数据增强方法，该方法从纯文本指令数据中合成类似视频的样本。将这些合成样本与视频数据混合，可以实现更高效的训练方案。通过全面的实验，我们证明了所提出的方法能够取得与使用显著更多样本训练的基础模型相当甚至更优的性能。同时，我们发现引入这些合成样本可以增强长视频理解能力，而无需在长视频数据上进行训练。代码和数据示例可在 https://github.com/VITA-MLLM/Sparrow 获取。
英文摘要：
Recent years have seen the success of Multimodal Large Language Models (MLLMs) in the domain of vision understanding. The success of these models can largely be attributed to the dominant scaling law, which states that larger parameter sizes and data volumes contribute to better performance. Notably, data scaling has been primarily driven by automatic data pipelines, which focus on the self-instruction of LLMs. The paradigm has been taken for granted for quite some time, but the study of the effectiveness of scaling with these data has been neglected for a long time. In this context, this work revisits scaling with synthetic data and focuses on developing video-LLMs from a data-centric perspective. Our primary study approach involves fine-tuning pre-trained image-LLMs with video data and examining learning efficiency through data scaling. Results from our preliminary experiments reveal a low learning efficiency phenomenon when simply scaling up video data samples, which, through our probing, can be ascribed to a lack of instruction diversity. Aiming at this issue, we propose a data augmentation method called Sparrow, which synthesizes video-like samples from pure text instruction data. Mixing these synthetic samples with the video data enables a more efficient training scheme. Through comprehensive experiments, we demonstrate that our proposed method achieves performance comparable to or even superior to that of baselines trained with significantly more samples. Meanwhile, we find that incorporating these synthetic samples can enhance the performance of long video understanding without requiring training on long video data. The code and data examples are available at https://github.com/VITA-MLLM/Sparrow.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
