arXiv 2503.180652025-11-04cs.CVcs.AIcs.CLcs.RO

从已见重写未见：使用基础模型重写观测-指令以增强视觉语言导航

Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation

Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；Hunan Artificial Intelligence and Robotics Institute Company Ltd.（湖南人工智能与机器人研究院有限公司）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Peng Cheng Laboratory（鹏城实验室）

Ziming Wei, Bingqian Lin, Yunshuang Nie, Jiaqi Chen, Shikui Ma, Hang Xu, Xiaodan Liang

备注：: Accepted by IEEE Transactions on Neural Networks and Learning Systems
DOI：: 10.1109/TNNLS.2025.3624691

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉语言导航数据稀缺问题，提出RAM范式，结合VLMs和LLMs重写并生成未见观测-指令对，配合混合训练策略，显著提升了智能体在未见环境中的泛化性能。

摘要：

数据稀缺是视觉语言导航（VLN）领域长期存在的挑战，极大阻碍了智能体对未见环境的泛化。以往工作主要依赖额外的模拟器数据或网络收集的图像/视频来提升泛化能力。然而，模拟器环境的多样性依然受限，且网络收集的数据通常需要大量人工去除噪声。本文提出一种用于VLN的重写驱动增强（RAM）范式，通过重写人工标注的训练数据，直接创建未见观测-指令对。得益于重写机制，无需模拟器且节省人工即可获取新的观测-指令对以促进泛化。具体而言，首先引入物体丰富观测重写，结合视觉语言模型（VLMs）和大语言模型（LLMs）推导出重写的物体丰富场景描述，通过文本到图像生成模型（T2IMs）实现具有多样物体和空间布局的观测合成。接着，提出观测对比指令重写，通过要求LLMs推理原始与新观测之间的差异，生成与观测对齐的重写指令。我们进一步开发了先混合后聚焦的训练策略与随机观测裁剪方案，在训练中有效增强数据分布多样性并抑制增强数据噪声。在离散环境（R2R、REVERIE和R4R数据集）与连续环境（R2R-CE数据集）上的实验表明，本方法性能优越且泛化能力出色。代码发布于 https://github.com/SaDil13/VLN-RAM。

英文摘要：

Data scarcity is a long-standing challenge in the Vision-Language Navigation (VLN) field, which extremely hinders the generalization of agents to unseen environments. Previous works primarily rely on additional simulator data or web-collected images/videos to improve the generalization. However, the simulator environments still face limited diversity, and the web-collected data often requires extensive labor to remove the noise. In this paper, we propose a Rewriting-driven AugMentation (RAM) paradigm for VLN, which directly creates the unseen observation-instruction pairs via rewriting human-annotated training data. Benefiting from our rewriting mechanism, new observation-instruction pairs can be obtained in both simulator-free and labor-saving manners to promote generalization. Specifically, we first introduce Object-Enriched Observation Rewriting, where we combine Vision-Language Models (VLMs) and Large Language Models (LLMs) to derive rewritten object-enriched scene descriptions, enabling observation synthesis with diverse objects and spatial layouts via Text-to-Image Generation Models (T2IMs). Then, we propose Observation-Contrast Instruction Rewriting, which generates observation-aligned rewritten instructions by requiring LLMs to reason the difference between original and new observations. We further develop a mixing-then-focusing training strategy with a random observation cropping scheme, effectively enhancing data distribution diversity while suppressing augmentation data noise during training. Experiments on both the discrete environments (R2R, REVERIE, and R4R datasets) and continuous environments (R2R-CE dataset) show the superior performance and impressive generalization ability of our method. Code is available at https://github.com/SaDil13/VLN-RAM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。