VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
Robotics Institute, Carnegie Mellon University(卡内基梅隆大学机器人研究所)
Haochen Zhang, Nader Zantout, Pujith Kachana, Zongyuan Wu, Ji Zhang, Wenshan Wang
备注:
Accepted and presented at the 1st Workshop on Semantic Reasoning and Goal Understanding in Robotics (SemRob), Robotics Science and Systems Conference (RSS 2024)
With the recent rise of Large Language Models (LLMs), Vision-Language Models (VLMs), and other general foundation models, there is growing potential for multimodal, multi-task embodied agents that can operate in diverse environments given only natural language as input. One such application area is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the spatial reasoning and semantic understanding required, particularly in arbitrary scenes that may contain many objects belonging to fine-grained classes. To address this challenge, we curate the largest real-world dataset for Vision and Language-guided Action in 3D Scenes (VLA-3D), consisting of over 11.5K scanned 3D indoor rooms from existing datasets, 23.5M heuristically generated semantic relations between objects, and 9.7M synthetically generated referential statements. Our dataset consists of processed 3D point clouds, semantic object and room annotations, scene graphs, navigable free space annotations, and referential language statements that specifically focus on view-independent spatial relations for disambiguating objects. The goal of these features is to aid the downstream task of navigation, especially on real-world systems where some level of robustness must be guaranteed in an open world of changing scenes and imperfect language. We benchmark our dataset with current state-of-the-art models to obtain a performance baseline. All code to generate and visualize the dataset is publicly released, see https://github.com/HaochenZ11/VLA-3D. With the release of this dataset, we hope to provide a resource for progress in semantic 3D scene understanding that is robust to changes and one which will aid the development of interactive indoor navigation systems.
CrossTracer: Cross-Embodiment Navigation via VLA Model Reasoning and Trace Residuals Adapting
Peng Cheng Laboratory(鹏城实验室);Southern University of Science and Technology(南方科技大学);Innovation Investment Research Institute(创新投资研究院);Soochow University(苏州大学)
From Local Whole-Body VLA Behaviors to Scene-Scale Aerial Manipulation
Nanyang Technological University(南洋理工大学);Tongji University(同济大学);Shanghai Institute of Intelligent Science and Technology(上海自主智能无人系统科学中心);Beihang University(北京航空航天大学);VinUniversity
Team Xiaomi EV-AD VLA: Caption-Guided Retrieval System for Cross-Modal Drone Navigation -- Technical Report for IROS 2025 RoboSense Challenge Track 4
Tsinghua University(清华大学);Xiaomi EV(小米电动车);Georgia Institute of Technology(佐治亚理工学院);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Renmin University of China(中国人民大学)
BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation
The Chinese University of Hong Kong(香港中文大学);The Third Affiliated Hospital of Sun Yat-sen University(中山大学附属第三医院);University of Cambridge(剑桥大学);University of California, Davis(加州大学戴维斯分校)
WareFly-VLA: A Vision-Language-Action Framework for UAV Navigation and Human Tracking in Smart Warehouses
Center for AI Research, VinUniversity(VinUniversity人工智能研究中心);College of Engineering and Computer Science, VinUniversity(VinUniversity工程与计算机科学学院);VinRobotics(VinRobotics公司)
Splat2Act: From Captured Scenes to Personalized VLA Manipulation
Institute for Robotics and Intelligent Machines, Georgia Institute of Technology(佐治亚理工学院机器人与智能机器研究所);Waymo LLC(Waymo公司);Toyota Research Institute(丰田研究所)
WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN
Institute of Artificial Intelligence, China Telecom(中国电信人工智能研究院);Zhejiang University(浙江大学);Tongji University(同济大学);Shanghai Jiao Tong University(上海交通大学)
HyperDCM: Dynamic Cluster Memory Replay in Hyperbolic Space for Continual Robotic Navigation Across Scenes
Software Engineering Institute, East China Normal University(华东师范大学软件工程学院);School of Geospatial Information, Information Engineering University(信息工程大学地理空间信息学院);University of Shanghai for Science and Technology(上海理工大学);Shanghai Jiao Tong University(上海交通大学)
NavTuner: Learning a Scene-Sensitive Family of Navigation Policies
College of Computing, Georgia Institute of Technology(佐治亚理工学院计算学院);Institute for Robotics and Intelligent Machines, Georgia Institute of Technology(佐治亚理工学院机器人与智能机器研究所)
Learning Actions for Drift-Free Navigation in Highly Dynamic Scenes
IIIT-Hyderabad(国际信息技术研究所海得拉巴分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
我们着手一个迄今未报告的问题,自主机器人(自动驾驶汽车)在动态场景中导航的方式,减少其定位误差和最终的累积漂移或绝对轨迹误差,这是在这样的动态场景中明显。以非常受欢迎的Velodyne-16 3D LIDAR作为主要的传感模式,以及精确的基于LIDAR的定位和映射算法LOAM作为状态估计框架,我们表明,在没有导航策略的情况下,漂移在移动物体的存在下迅速积累。为了克服这一点,我们通过一组合适的奖励和惩罚函数来学习导致漂移最小化导航的动作。我们使用近端策略优化,一类深度强化学习方法,来学习导致漂移最小化轨迹的动作。我们通过对各种合成的,但照片般逼真的场景通过CARLA模拟器提供的优越性能的建议框架相对于不采用这种政策的方法进行了广泛的比较。