VG-TVP：基于视觉锚定的文本-视频提示的多模态程序规划
VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting
AI总结：
提出VG-TVP框架，利用LLM、视频字幕和扩散模型，通过FoC及双向桥接生成多模态程序计划，并在新数据集Daily-PP上超越单模态基线。
摘要：
基于大型语言模型（LLM）的智能体在程序性任务中展现出潜力，但由文本和视频增强的多模态指令辅助用户的潜力仍未得到充分探索。为解决这一空白，我们提出了视觉锚定的文本-视频提示（VG-TVP）方法，这是一种新颖的基于LLM的多模态程序规划（MPP）框架。给定一个高层目标，它能生成连贯的文本和视频程序计划。主要挑战在于实现程序计划中的文本和视觉信息丰富性、时间连贯性以及准确性。VG-TVP利用了LLM的零样本推理能力、视频字幕模型的视频到文本生成能力以及扩散模型的文本到视频生成能力。通过提出新颖的字幕融合（FoC）方法，并使用文本到视频桥（T2V-B）和视频到文本桥（V2T-B），VG-TVP改善了模态间的交互。这些桥接机制使LLM能够指导生成视觉锚定的文本计划和文本锚定的视频计划。为解决适合MPP的数据集稀缺问题，我们构建了一个新数据集，名为日常生活任务程序计划（Daily-PP）。我们进行了全面的实验和基准测试，以评估人类偏好（涉及文本和视觉信息丰富性、时间连贯性和计划准确性）。我们的VG-TVP方法在Daily-PP数据集上优于单模态基线方法。
英文摘要：
Large Language Model (LLM)-based agents have shown promise in procedural tasks, but the potential of multimodal instructions augmented by texts and videos to assist users remains under-explored. To address this gap, we propose the Visually Grounded Text-Video Prompting (VG-TVP) method which is a novel LLM-empowered Multimodal Procedural Planning (MPP) framework. It generates cohesive text and video procedural plans given a specified high-level objective. The main challenges are achieving textual and visual informativeness, temporal coherence, and accuracy in procedural plans. VG-TVP leverages the zero-shot reasoning capability of LLMs, the video-to-text generation ability of the video captioning models, and the text-to-video generation ability of diffusion models. VG-TVP improves the interaction between modalities by proposing a novel Fusion of Captioning (FoC) method and using Text-to-Video Bridge (T2V-B) and Video-to-Text Bridge (V2T-B). They allow LLMs to guide the generation of visually-grounded text plans and textual-grounded video plans. To address the scarcity of datasets suitable for MPP, we have curated a new dataset called Daily-Life Task Procedural Plans (Daily-PP). We conduct comprehensive experiments and benchmarks to evaluate human preferences (regarding textual and visual informativeness, temporal coherence, and plan accuracy). Our VG-TVP method outperforms unimodal baselines on the Daily-PP dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
