arXiv 2502.180412026-03-01cs.CVcs.RO

Openfly：面向空中视觉-语言导航的综合性平台

Openfly: A comprehensive platform for aerial vision-language navigation

Shanghai AI Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Beihang University（北航）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；East China University of Science and Technology（东华大学）；Fudan University（复旦大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；TeleAI

Yunpeng Gao, Chenhui Li, Zhongrui You, Junli Liu, Zhen Li, Pengan Chen, Qizhi Chen, Zhonghan Tang, Liansheng Wang, Penghui Yang, Yiwen Tang, Yuhang Tang, Shuai … 展开作者

Yunpeng Gao, Chenhui Li, Zhongrui You, Junli Liu, Zhen Li, Pengan Chen, Qizhi Chen, Zhonghan Tang, Liansheng Wang, Penghui Yang, Yiwen Tang, Yuhang Tang, Shuai Liang, Songyi Zhu, Ziqin Xiong, Yifei Su, Xinyi Ye, Jianan Li, Yan Ding, Dong Wang, Xuelong Li, Zhigang Wang, Bin Zhao

备注：: accepted by ICLR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

OpenFly平台通过整合多种渲染引擎和自动化工具链，构建大规模空中VLN数据集，并提出关键帧感知的VLN模型，提升户外空中视觉-语言导航的研究与应用。

摘要：

视觉-语言导航（VLN）旨在通过利用语言指令和视觉线索来引导智能体，其在具身人工智能中扮演着关键角色。室内VLN已得到广泛研究，而户外空中VLN仍处于探索阶段。潜在的原因是户外空中视角涵盖广阔的区域，使数据收集更加困难，导致缺乏基准测试。为了解决这个问题，我们提出了OpenFly平台，包括多种渲染引擎、多功能工具链和大规模的空中VLN基准测试。首先，我们整合了多种渲染引擎和先进的环境模拟技术，包括Unreal Engine、GTA V、Google Earth和3D高斯点散射（3D GS）。特别是，3D GS支持实拍到模拟的渲染，进一步增强了我们环境的真实感。其次，我们开发了一个高度自动化的工具链，用于空中VLN数据的收集，简化了点云采集、场景语义分割、飞行轨迹创建和指令生成。第三，基于该工具链，我们构建了一个大规模的空中VLN数据集，包含10万条轨迹，覆盖了18个场景中的不同高度和长度。此外，我们提出了OpenFly-Agent，一个关键帧感知的VLN模型，强调飞行过程中的关键观察。为了基准测试，进行了广泛的实验和分析，评估了几种最近的VLN方法，并展示了我们OpenFly平台和代理的优越性。工具链、数据集和代码将开源。

英文摘要：

Vision-Language Navigation (VLN) aims to guide agents by leveraging language instructions and visual cues, playing a pivotal role in embodied AI. Indoor VLN has been extensively studied, whereas outdoor aerial VLN remains underexplored. The potential reason is that outdoor aerial view encompasses vast areas, making data collection more challenging, which results in a lack of benchmarks. To address this problem, we propose OpenFly, a platform comprising various rendering engines, a versatile toolchain, and a large-scale benchmark for aerial VLN. Firstly, we integrate diverse rendering engines and advanced techniques for environment simulation, including Unreal Engine, GTA V, Google Earth, and 3D Gaussian Splatting (3D GS). Particularly, 3D GS supports real-to-sim rendering, further enhancing the realism of our environments. Secondly, we develop a highly automated toolchain for aerial VLN data collection, streamlining point cloud acquisition, scene semantic segmentation, flight trajectory creation, and instruction generation. Thirdly, based on the toolchain, we construct a large-scale aerial VLN dataset with 100k trajectories, covering diverse heights and lengths across 18 scenes. Moreover, we propose OpenFly-Agent, a keyframe-aware VLN model emphasizing key observations during flight. For benchmarking, extensive experiments and analyses are conducted, evaluating several recent VLN methods and showcasing the superiority of our OpenFly platform and agent. The toolchain, dataset, and codes will be open-sourced.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。