流式干预：视频大语言模型能否在错误发生时即时纠正？
Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
AI总结：
提出Ego-MC-Bench基准评估视频LLM在烹饪场景中的实时干预能力，并构建Ego-CoMist反事实合成数据集提升小模型性能。
摘要：
学习日常技能（如烹饪一道菜）越来越依赖于教学媒体，例如在线视频。这为使用视频（和多模态）大语言模型（LLMs）作为任务指导助手打开了大门。一个潜在的任务指导助手在现实世界中成功的关键能力是，它能够在错误一出现时就主动干预以引导用户。为了评估这一关键能力，我们引入了Ego-MC-Bench（错误纠正），这是一个用于评估在现实烹饪场景中反应性、逐步任务指导的基准。大量实验表明，Ego-MC-Bench对于最先进的视频LLMs具有高度挑战性。我们认为一个关键原因是用于在此任务上微调模型的训练数据有限。尽管存在广泛的烹饪视频数据集，但现有数据集缺乏错误示例以及适当时间的干预。为了帮助解决这一数据限制，我们还引入了Ego-CoMist，这是一个反事实合成数据集，通过将非交互式烹饪视频转换为显示主动干预的监督训练示例而创建。我们表明，在Ego-CoMist上进行微调可以带来性能提升，特别是对于更适合在边缘设备上提供帮助的更小、更高效的视频LLMs。
英文摘要：
Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) large language models (LLMs) as task guidance assistants. A crucial capability for the real-world success of a prospective task guidance assistant is it's ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a benchmark for evaluating reactive, step-by-step task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state-of-the-art video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking video datasets, existing datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic dataset created by transforming non -interactive cooking videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient video LLMs that are well suited for delivering assistance on edge devices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
