计划、观察、恢复：主动式程序辅助的基准与架构
Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance
AI总结：
提出EgoProactive数据集和Pro²Bench基准，并设计解耦规划器-交互架构，用于主动式程序辅助中的实时引导和异常恢复。
摘要：
我们设想一个主动的多模态辅助系统，该系统在程序性任务中为用户提供实时的逐步指导，自主决定何时中断以及如何指导。然而，由于缺乏反映现实条件的大规模跨领域基准，特别是用户偏离预期步骤序列的常见情况，进展受到限制。我们通过四项贡献来解决这一差距： extbf{(1)}~我们发布了 extbf{EgoProactive}，一个大规模的可穿戴自我中心数据集，用于主动程序辅助，带有明确的计划外（OOP）标注和恢复步骤； extbf{(2)}~我们将五个已建立的基准（Ego4D、EPIC-KITCHENS、EgoExo4D、HoloAssist、HowTo100M）扩充为统一的主动指导模式下的 extbf{Pro extsuperscript{2}Bench}； extbf{(3)}~我们提出了一种专门针对程序状态、视觉线索和恢复注入的 extbf{解耦规划器-交互架构}； extbf{(4)}~我们引入了一种跨模型家族迁移的训练后方案，通过在Llama~4和Qwen-3.6-VL上的跨骨干复制进行验证。在大量实验中，我们训练的Llama-4系统在所有六个数据集上，相对于强大的专有基线（Claude Opus~4.6、Gemini~3.1~Pro、GPT~5.2）和开放权重基线（Qwen3~VL~235B），显著提高了客观干预质量。Oracle计划实验进一步表明，当计划质量得到控制时，训练的双工模型产生高质量的指导，并在计划外恢复方面取得巨大收益。
英文摘要：
We envision a proactive multi-modal assistant system which gives users real-time step-by-step guidance on a procedural task, autonomously deciding \textit{when} to interrupt, and \textit{how} to coach. However, progress is limited by the absence of large-scale, cross-domain benchmarks that reflect realistic conditions, particularly the common case in which users deviate from the expected step sequence. We address this gap with four contributions: \textbf{(1)}~we release \textbf{EgoProactive}, a large-scale wearable-egocentric dataset for proactive procedural assistance with explicit Out-of-Plan (OOP) annotations and recovery steps; \textbf{(2)}~we augment five established benchmarks (Ego4D, EPIC-KITCHENS, EgoExo4D, HoloAssist, HowTo100M) into \textbf{Pro\textsuperscript{2}Bench} under a unified proactive-guidance schema; \textbf{(3)}~we propose a \textbf{decoupled planner--interaction architecture} specialized for procedural state, visual cues, and recovery injection; \textbf{(4)}~we introduce a post-training recipe that transfers across model families, validated by cross-backbone replication on Llama~4 and Qwen-3.6-VL. In extensive experiments, our trained Llama-4 system substantially improves objective intervention quality over strong proprietary baselines (Claude Opus~4.6, Gemini~3.1~Pro, GPT~5.2) and open-weight baselines (Qwen3~VL~235B) baselines across all six datasets. Oracle-plan experiments further show that, when plan quality is controlled, the trained duplex model produces high-quality guidance and large gains on Out-of-Plan recovery.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
