提出、评估、搜索：利用大语言模型实现教学视频中的目标导向规划
Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos
AI总结：
本文提出VidAssist框架，利用大语言模型作为知识库和评估工具，结合广度优先搜索算法，在零样本和少样本条件下实现教学视频中的目标导向规划，并在COIN数据集上超越全监督方法。
摘要：
目标导向规划，即预测一系列动作使智能体从当前状态过渡到预定义目标，对于开发辅助用户完成日常程序性任务的智能助手至关重要。该问题面临重大挑战，因为需要全面了解时间和层次化任务结构，以及强大的推理和规划能力。为实现这一目标，先前的工作通常依赖于在目标数据集上进行大量训练，这往往导致显著的数据集偏差和对未见任务缺乏泛化能力。在这项工作中，我们提出了VidAssist，这是一个集成框架，专为教学视频中的零样本/少样本目标导向规划而设计。VidAssist利用大语言模型（LLMs）作为知识库和评估工具，用于生成和评估动作计划，从而克服了从小规模、低多样性数据集中获取程序性知识的挑战。此外，VidAssist采用广度优先搜索算法进行最优计划生成，其中利用为目标导向规划设计的价值函数组合来评估每一步预测的动作。大量实验表明，VidAssist为不同的目标导向规划设置（例如视觉辅助规划（VPA）和程序规划（PP））提供了统一框架，并在零样本和少样本设置中取得了显著性能。具体而言，在COIN数据集上预测4个未来动作时，我们的少样本模型在VPA任务上比先前全监督的最先进方法高出+7.7%，在PP任务上高出+4.81%。代码和模型已在https://sites.google.com/view/vidassist公开可用。
英文摘要：
Goal-oriented planning, or anticipating a series of actions that transition an agent from its current state to a predefined objective, is crucial for developing intelligent assistants aiding users in daily procedural tasks. The problem presents significant challenges due to the need for comprehensive knowledge of temporal and hierarchical task structures, as well as strong capabilities in reasoning and planning. To achieve this, prior work typically relies on extensive training on the target dataset, which often results in significant dataset bias and a lack of generalization to unseen tasks. In this work, we introduce VidAssist, an integrated framework designed for zero/few-shot goal-oriented planning in instructional videos. VidAssist leverages large language models (LLMs) as both the knowledge base and the assessment tool for generating and evaluating action plans, thus overcoming the challenges of acquiring procedural knowledge from small-scale, low-diversity datasets. Moreover, VidAssist employs a breadth-first search algorithm for optimal plan generation, in which a composite of value functions designed for goal-oriented planning is utilized to assess the predicted actions at each step. Extensive experiments demonstrate that VidAssist offers a unified framework for different goal-oriented planning setups, e.g., visual planning for assistance (VPA) and procedural planning (PP), and achieves remarkable performance in zero-shot and few-shot setups. Specifically, our few-shot model outperforms the prior fully supervised state-of-the-art method by +7.7% in VPA and +4.81% PP task on the COIN dataset while predicting 4 future actions. Code, and models are publicly available at https://sites.google.com/view/vidassist.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
