RECIPE: 通过指令视频中的 grounding 实现过程规划
RECIPE: Procedural Planning via Grounding in Instructional Video
AI总结：
该研究提出RECIPE方法，通过利用指令视频中的grounding信息来改进过程规划任务，通过利用预计算的文本嵌入实现大规模视频数据的验证，从而提升规划的准确性和鲁棒性。
摘要：
视觉规划要求模型在给定部分视频上下文和目标的情况下，生成剩余步骤的自然语言描述。该任务的进展受到标注的限制：干净的标记数据集较小，领域狭窄，每个示例只编码一个执行轨迹，尽管许多有效的顺序存在。大规模的指令视频语料库提供了数量级更多的过程内容，但通过使用伪标签进行监督微调会传播分割和对齐错误，并且只能生成单轨迹。我们识别出一个关键的不对称性：从噪声视频中提取干净的步骤标签是困难的，但验证生成的步骤序列是否在ASR转录中时间上接地是便宜的，并且可以通过预计算的文本嵌入扩展到数百万个视频。我们利用这种不对称性，在RECIPE中将grounding质量作为GRPO的奖励，将噪声语料库转化为验证者而不是标签来源。该框架可以统一应用于两种规划器输入配置（Socratic，使用冻结的VLM提取文本历史，以及Video，直接消耗视频令牌）以及标注和弱监督的模式。我们在7个过程基准上进行评估，使用基于参考的LLM-as-judge协议对计划进行评分，跨6个过程标准。RECIPE-RL在所有规模（0.5B、3B、7B）和每个基准上都优于基础检查点，领域内宏准确率提升7到8分，在零样本情况下最高提升16分。它在标注和伪标签计划上均优于监督微调（后者会降低基础模型性能），并在没有人工标注的情况下保持稳健。作为先前提案-评估-搜索规划器的提案阶段使用时，在视觉规划辅助任务中在每个时间范围内均优于最强的零样本基线，在COIN任务中保持了SFT所崩溃的生成多样性。
英文摘要：
Visual planning asks a model to generate the remaining steps of a procedure in natural language given a partial video context and a goal. Progress on this task is bottlenecked by annotation: clean labeled datasets are small, domain-narrow, and encode a single execution trajectory per example, even though many valid orderings exist. Large-scale instructional video corpora offer orders of magnitude more procedural content, but supervised fine-tuning on pseudo-labels from their noisy ASR narrations propagates segmentation and alignment errors and stays single-trajectory. We identify a key asymmetry: extracting clean step labels from noisy video is hard, but verifying whether a generated step sequence is temporally grounded in ASR transcripts is cheap and scales to millions of videos via precomputed text embeddings. We exploit this asymmetry in RECIPE, which uses grounding quality as a reward for GRPO, turning the noisy corpus into a verifier rather than a label source. The framework applies uniformly to two planner input configurations (Socratic, with a textual history extracted by a frozen VLM, and Video, consuming video tokens directly) and to annotated and weakly supervised regimes. We evaluate on 7 procedural benchmarks using a reference-based LLM-as-judge protocol scoring plans across 6 procedural criteria. RECIPE-RL improves over the base checkpoint at all scales (0.5B, 3B, 7B) and every benchmark, with macro-accuracy gains of +7 to +8 points in-domain and up to +16 points zero-shot. It outperforms supervised fine-tuning on both annotated and pseudo-labeled plans (the latter degrades the base) and remains robust without human annotations. Used as the proposal stage of a prior propose-assess-search planner, it improves over the strongest zero-shot baseline at every horizon on Visual Planning for Assistance, and on COIN it preserves the generation diversity that SFT collapses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
