InstruGen：基于大型多模态模型的视觉语言导航自动指令生成
InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models
AI总结：
针对视觉语言导航智能体泛化差的问题，提出 InstruGen 范式，利用大型多模态模型从 YouTube 视频自动生成高质量路径-指令对，并通过多阶段验证减少幻觉，在 R2R 和 RxR 基准上达到最先进性能。
摘要：
近期关于视觉语言导航（VLN）的研究表明，由于缺乏真实的训练环境和高质量的路径-指令对，智能体在未见过的环境中泛化能力较差。现有大多数构建真实导航场景的方法成本较高，且指令的扩展主要依赖预定义的模板或规则，缺乏适应性。为缓解该问题，我们提出了 InstruGen，一种 VLN 路径-指令对生成范式。具体而言，我们使用 YouTube 房屋导览视频作为真实导航场景，并利用大型多模态模型（LMMs）强大的视觉理解与生成能力，自动生成多样且高质量的 VLN 路径-指令对。我们的方法生成不同粒度的导航指令，并实现指令与视觉观察之间的细粒度对齐，这是以往方法难以实现的。此外，我们设计了多阶段验证机制，以减少 LMMs 的幻觉和不一致性。实验结果表明，使用 InstruGen 生成的路径-指令对训练的智能体在 R2R 和 RxR 基准上取得了最先进的性能，尤其是在未见过的环境中。代码可在 https://github.com/yanyu0526/InstruGen 获取。
英文摘要：
Recent research on Vision-and-Language Navigation (VLN) indicates that agents suffer from poor generalization in unseen environments due to the lack of realistic training environments and high-quality path-instruction pairs. Most existing methods for constructing realistic navigation scenes have high costs, and the extension of instructions mainly relies on predefined templates or rules, lacking adaptability. To alleviate the issue, we propose InstruGen, a VLN path-instruction pairs generation paradigm. Specifically, we use YouTube house tour videos as realistic navigation scenes and leverage the powerful visual understanding and generation abilities of large multimodal models (LMMs) to automatically generate diverse and high-quality VLN path-instruction pairs. Our method generates navigation instructions with different granularities and achieves fine-grained alignment between instructions and visual observations, which was difficult to achieve with previous methods. Additionally, we design a multi-stage verification mechanism to reduce hallucinations and inconsistency of LMMs. Experimental results demonstrate that agents trained with path-instruction pairs generated by InstruGen achieves state-of-the-art performance on the R2R and RxR benchmarks, particularly in unseen environments. Code is available at https://github.com/yanyu0526/InstruGen.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
