SLVMBench：从视频记忆中学习技能
SLVMBench: Skill Learning from Video Memory
- Tsinghua University（清华大学）
- University of Cambridge（剑桥大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
SLVMBench是首个用于评估视频大语言模型从长视频记忆学习技能并应用于实时任务能力的基准测试，通过特定视频流和人工标注进行测试，发现现有视频LLMs在此方面有局限。
AI中文摘要：
我们介绍了从视频记忆中学习技能（SLVMBench），这是首个联合评估视频大语言模型（video-LLMs）能否从长视频记忆中学习技能并将其应用于实时任务的基准测试。SLVMBench为模型提供2至3小时的视频流，其中包含嵌入在任意无关视频流中的教程视频，类似现实世界中的人类学习实践。要求Video-LLMs应用所学技能回答关于正在播放视频的实时问题。与强调被动理解的长视频理解基准测试和依赖简短即时演示的技能学习基准测试不同，SLVMBench测试记忆和提取程序知识以及将其转移到实时任务的完整流程。此外，严格的人工标注具有亚秒级时间校准、消除常识猜测的人工设计问题以及整理的教程，以确保所需技能的覆盖范围。对现有先进的专有和开源视频LLMs的评估表明，视频LLMs在从视频中学习和应用技能知识方面存在很大困难。而且，当技能知识置于长视频记忆中时，性能会显著下降。这些结果揭示了现有视频LLMs的一个关键局限性，并将SLVMBench定位为研究从长上下文视频记忆中进行实时技能获取和应用的首个基准测试。
英文摘要：
We introduce Skill Learning from Video Memory (SLVMBench), the first benchmark that jointly evaluates whether video large language models (video-LLMs) can learn skills from long video memory and apply them to real-time tasks. SLVMBench presents models with 2-3 hour video streams that contain a tutorial video embedded in a stream of arbitrary irrelevant videos, resembling real-world human learning practices. Video-LLMs are asked to apply the acquired skill to answer real-time questions about an ongoing video. Unlike long-video understanding benchmarks that emphasize passive comprehension and skill-learning benchmarks that rely on short, immediate demonstrations, SLVMBench tests the full pipeline of memorizing and extracting procedural knowledge, as well as transferring it to real-time tasks. Moreover, rigorous human annotations feature sub-second-level temporal calibration, manually engineered questions eliminating common-sense guessing, and collated tutorials to ensure coverage of the required skills. Evaluations on state-of-the-art proprietary and open-source video LLMs show that video-LLMs struggle substantially with learning and applying skill knowledge from videos. Moreover, performance degrades markedly when the skill knowledge is placed within a long video memory. These results reveal a key limitation of existing video LLMs and position SLVMBench as the first benchmark for studying real-time skill acquisition and application from long-context video memory.
