通过精心设计的数据分布特性在视频视觉语言模型中激发上下文学习能力
Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties
AI总结：
本文提出EILEV训练范式，通过捕捉突发性分布、偏斜边际分布和动态语义等预训练数据关键特性，在视频视觉语言模型中诱导上下文学习能力，并在少样本视频叙述任务上优于现有VLMs。
摘要：
大型语言模型（LLMs）近期取得成功的一个重要原因是其上下文学习能力，这使得可以通过用少量相关示例进行提示，将其快速适配到下游基于文本的任务。虽然近年来已经开发出用于同时需要文本和图像的任务的大型视觉语言模型（VLMs），但它们在视觉信息上的上下文学习能力仍然十分欠缺，尤其是在理解和生成关于视频的文本方面。在这项工作中，我们实现了视频上的涌现式上下文学习（Emergent In-context Learning on Videos，EILEV），这是一种新颖的训练范式，通过捕捉先前工作发现的、对Transformer中的上下文学习至关重要的预训练数据的关键特性，来诱导视频和文本上的上下文学习。在我们的实验中，我们表明，对于新颖、罕见的动作，EILEV训练的模型在少样本视频叙述方面优于其他现成的VLMs。此外，我们证明，突发性分布、偏斜边际分布和动态语义这些关键特性，在叙述程序性视频时，对VLMs的上下文学习能力各自有不同程度的贡献。我们的结果、分析和EILEV训练的模型为视频和文本上上下文学习的涌现提供了大量见解，为未来优化和扩展VLMs以进行开放域视频理解和推理奠定了基础。我们的代码和演示可在 https://github.com/yukw777/EILEV 获取。
英文摘要：
A major reason behind the recent success of large language models (LLMs) is their \textit{in-context learning} capability, which makes it possible to rapidly adapt them to downstream text-based tasks by prompting them with a small number of relevant demonstrations. While large vision-language models (VLMs) have recently been developed for tasks requiring both text and images, they largely lack in-context learning over visual information, especially in understanding and generating text about videos. In this work, we implement \textbf{E}mergent \textbf{I}n-context \textbf{Le}arning on \textbf{V}ideos (\eilev{}), a novel training paradigm that induces in-context learning over video and text by capturing key properties of pre-training data found by prior work to be essential for in-context learning in transformers. In our experiments, we show that \eilev-trained models outperform other off-the-shelf VLMs in few-shot video narration for novel, rare actions. Furthermore, we demonstrate that these key properties of bursty distributions, skewed marginal distributions, and dynamic meaning each contribute to varying degrees to VLMs' in-context learning capability in narrating procedural videos. Our results, analysis, and \eilev{}-trained models yield numerous insights about the emergence of in-context learning over video and text, creating a foundation for future work to optimize and scale VLMs for open-domain video understanding and reasoning. Our code and demo are available at \url{https://github.com/yukw777/EILEV}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
