Demo-ICL: 用于过程视频知识获取的上下文学习
Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition
AI总结：
Demo-ICL通过两阶段训练策略提升模型从上下文示例中学习的能力，针对视频理解中的新挑战提出Demo-ICL-Bench基准测试。
摘要：
尽管近年来多模态大语言模型（MLLMs）在视频理解方面的能力不断增强，但现有的视频基准测试主要评估模型静态内部知识的理解能力，而非其从少量新动态上下文中学习和适应的能力。为弥合这一差距，我们提出了Demo驱动的视频上下文学习，这是一个专注于通过上下文演示来回答目标视频问题的新任务。同时，我们提出了Demo-ICL-Bench，一个具有挑战性的基准测试，旨在评估演示驱动的视频上下文学习能力。Demo-ICL-Bench由1200个带有相关问题的教育YouTube视频构建而成，从中衍生出两种类型的演示：（i）总结视频字幕用于文本演示；（ii）对应的教育视频作为视频演示。为了有效应对这一新挑战，我们开发了Demo-ICL，一种具有两阶段训练策略的MLLM：视频监督微调和信息辅助的直接偏好优化，共同增强模型从上下文示例中学习的能力。大量实验表明，Demo-ICL-Bench的难度，验证了Demo-ICL的有效性，并揭示了未来研究方向。
英文摘要：
Despite the growing video understanding capabilities of recent Multimodal Large Language Models (MLLMs), existing video benchmarks primarily assess understanding based on models' static, internal knowledge, rather than their ability to learn and adapt from dynamic, novel contexts from few examples. To bridge this gap, we present Demo-driven Video In-Context Learning, a novel task focused on learning from in-context demonstrations to answer questions about the target videos. Alongside this, we propose Demo-ICL-Bench, a challenging benchmark designed to evaluate demo-driven video in-context learning capabilities. Demo-ICL-Bench is constructed from 1200 instructional YouTube videos with associated questions, from which two types of demonstrations are derived: (i) summarizing video subtitles for text demonstration; and (ii) corresponding instructional videos as video demonstrations. To effectively tackle this new challenge, we develop Demo-ICL, an MLLM with a two-stage training strategy: video-supervised fine-tuning and information-assisted direct preference optimization, jointly enhancing the model's ability to learn from in-context examples. Extensive experiments with state-of-the-art MLLMs confirm the difficulty of Demo-ICL-Bench, demonstrate the effectiveness of Demo-ICL, and thereby unveil future research directions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
