LAP：一种语言感知的规划模型用于教学视频中的过程规划
LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos
AI总结：
LAP通过利用语言描述提升过程规划的准确性，实现教学视频中动作序列的高效生成。
摘要：
过程规划需要一个模型预测将起始视觉观察转换为目标的一系列动作，在教学视频中。尽管大多数现有方法主要依赖视觉观察作为输入，但它们在本质上存在模糊性，其中不同的动作可能在视觉上相似。在本工作中，我们主张语言描述在潜在空间中提供了更独特的表示，用于过程规划。我们引入了语言感知规划（LAP），一种新颖的方法，利用语言的表达力来弥合视觉观察和规划之间的差距。LAP使用微调的视觉语言模型（VLM）将视觉观察转换为文本描述，并预测动作和提取文本嵌入。这些文本嵌入比视觉嵌入更具区分性，并用于扩散模型中进行动作序列的规划。我们评估LAP在三个过程规划基准上：CrossTask、Coin和NIV。LAP在多个指标和时间范围内实现了新的最先进的性能，证明了语言感知规划的显著优势。
英文摘要：
Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
