This paper focuses on visual semantic navigation, the task of producing actions for an active agent to navigate to a specified target object category in an unknown environment. To complete this task, the algorithm should simultaneously locate and navigate to an instance of the category. In comparison to the traditional point goal navigation, this task requires the agent to have a stronger contextual prior to indoor environments. We introduce SSCNav, an algorithm that explicitly models scene priors using a confidence-aware semantic scene completion module to complete the scene and guide the agent's navigation planning. Given a partial observation of the environment, SSCNav first infers a complete scene representation with semantic labels for the unobserved scene together with a confidence map associated with its own prediction. Then, a policy network infers the action from the scene completion result and confidence map. Our experiments demonstrate that the proposed scene completion module improves the efficiency of the downstream navigation policies. Video, code, and data: https://sscnav.cs.columbia.edu/
Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environments
New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；Robotics X, Tencent（腾讯Robotics X实验室）
We propose an approach for mapping natural language instructions and raw observations to continuous control of a quadcopter drone. Our model predicts interpretable position-visitation distributions indicating where the agent should go during execution and where it should stop, and uses the predicted distributions to select the actions to execute. This two-step model decomposition allows for simple and efficient training using a combination of supervised learning and imitation learning. We evaluate our approach with a realistic drone simulator, and demonstrate absolute task-completion accuracy …
Improving Visual Place Recognition Based Robot Navigation By Verifying Localization Estimates
Queensland University of Technology（昆士兰科技大学）；QUT Centre for Robotics（昆士兰科技大学机器人中心）；School of Electrical Engineering and Robotics（电气工程与机器人学院）；Australian National University（澳大利亚国立大学）；College of Engineering, Computing and Cybernetics（工程、计算与网络学院）
Many robotic tasks involving some form of 3D visual perception greatly benefit from a complete knowledge of the working environment. However, robots often have to tackle unstructured environments and their onboard visual sensors can only provide incomplete information due to limited workspaces, clutter or object self-occlusion. In recent years, deep learning architectures for shape completion have begun taking traction as effective means of inferring a complete 3D object representation from partial visual data. Nevertheless, most of the existing state-of-the-art approaches provide a fixed outp…
Distilling LLM Prior to Flow Model for Generalizable Agent's Imagination in Object Goal Navigation
Sun Yat-sen University（中山大学）；The University of Hong Kong（香港大学）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与高级计算重点实验室）
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Samsung Research（三星研究院）；South China Normal University（华南师范大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Depth completion aims at predicting dense pixel-wise depth from an extremely sparse map captured from a depth sensor, e.g., LiDARs. It plays an essential role in various applications such as autonomous driving, 3D reconstruction, augmented reality, and robot navigation. Recent successes on the task have been demonstrated and dominated by deep learning based solutions. In this article, for the first time, we provide a comprehensive literature review that helps readers better grasp the research trends and clearly understand the current advances. We investigate the related studies from the design…
Politecnico di Milano（米兰理工大学）；CNRS-Université de Strasbourg（法国国家科学研究中心-斯特拉斯堡大学）；Delft University of Technology（代尔夫特理工大学）；Scuola Superiore Sant’Anna（圣安娜高等学校）；European Institute of Oncology (IRCCS)（欧洲肿瘤研究所（IRCCS））
Navigation inside luminal organs is an arduous task that requires non-intuitive coordination between the movement of the operator's hand and the information obtained from the endoscopic video. The development of tools to automate certain tasks could alleviate the physical and mental load of doctors during interventions, allowing them to focus on diagnosis and decision-making tasks. In this paper, we present a synergic solution for intraluminal navigation consisting of a 3D printed endoscopic soft robot that can move safely inside luminal structures. Visual servoing, based on Convolutional Neur…
University of Adelaide（阿德莱德大学）；The Australian National University（澳大利亚国立大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
Vision-and-language navigation requires an agent to navigate through a real 3D environment following natural language instructions. Despite significant advances, few previous works are able to fully utilize the strong correspondence between the visual and textual sequences. Meanwhile, due to the lack of intermediate supervision, the agent's performance at following each part of the instruction cannot be assessed during navigation. In this work, we focus on the granularity of the visual and language sequences as well as the traceability of agents through the completion of an instruction. We pro…
Shape Completion and Real-Time Visualization in Robotic Ultrasound Spine Acquisitions
Computer Aided Medical Procedures, Technical University of Munich（技术大学慕尼黑计算机辅助医疗程序）；Konrad Zuse School of Excellence in Reliable AI (relAI)（Konrad Zuse卓越人工智能学校）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrapping
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；University of Adelaide（阿德莱德大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Nanjing University（南京大学）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology（哈尔滨工业大学）；Fudan University（复旦大学）
East China Normal University（华东师范大学）；University of Guelph（圭尔夫大学）；Ryerson University（瑞尔森大学）
For people who ardently love painting but unfortunately have visual impairments, holding a paintbrush to create a work is a very difficult task. People in this special group are eager to pick up the paintbrush, like Leonardo da Vinci, to create and make full use of their own talents. Therefore, to maximally bridge this gap, we propose a painting navigation system to assist blind people in painting and artistic creation. The proposed system is composed of cognitive system and guidance system. The system adopts drawing board positioning based on QR code, brush navigation based on target detectio…
FastMap: Real-Time Semantic Map Completion via Bitwise Masked Modeling
NYUAD Center for Artificial Intelligence and Robotics (CAIR)（纽约大学阿布扎克人工智能与机器人中心）；New York University Abu Dhabi（纽约大学阿布扎克分校）；New York University（纽约大学）；Embodied AI and Robotics (AIR) Lab（具身人工智能与机器人实验室）；School of Software, Tsinghua University（清华大学软件学院）
Robust geometric and semantic scene understanding is ever more important in many real-world applications such as autonomous driving and robotic navigation. In this paper, we propose a multi-task learning-based approach capable of jointly performing geometric and semantic scene understanding, namely depth prediction (monocular depth estimation and depth completion) and semantic scene segmentation. Within a single temporally constrained recurrent network, our approach uniquely takes advantage of a complex series of skip connections, adversarial training and the temporal constraint of sequential …
Heriot-Watt University（赫瑞-瓦特大学）；Amazon Alexa AI（亚马逊 Alexa AI）；University of Southern California（南加州大学）
Language-guided robots performing home and office tasks must navigate in and interact with the world. Grounding language instructions against visual observations and actions to take in an environment is an open challenge. We present Embodied BERT (EmBERT), a transformer-based model which can attend to high-dimensional, multi-modal inputs across long temporal horizons for language-conditioned task completion. Additionally, we bridge the gap between successful object-centric navigation models used for non-interactive agents and the language-guided visual task completion benchmark, ALFRED, by int…
Real-time scene reconstruction from depth data inevitably suffers from occlusion, thus leading to incomplete 3D models. Partial reconstructions, in turn, limit the performance of algorithms that leverage them for applications in the context of, e.g., augmented reality, robotic navigation, and 3D mapping. Most methods address this issue by predicting the missing geometry as an offline optimization, thus being incompatible with real-time applications. We propose a framework that ameliorates this issue by performing scene reconstruction and semantic scene completion jointly in an incremental and …
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
Mininglamp Technology（旷视科技）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所综合技术集成中心）
PointSSC: A Cooperative Vehicle-Infrastructure Point Cloud Benchmark for Semantic Scene Completion
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；Mogo Auto Intelligence and Telematics Information Technology Company Ltd.（摩智智行智能与远程信息技术有限公司）
IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals
Fraunhofer Institute IVI（弗劳恩霍夫研究所IVI）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of California, Los Angeles（加州大学洛杉矶分校）
V2VSSC: A 3D Semantic Scene Completion Benchmark for Perception with Vehicle to Vehicle Communication
Nanjing University of Information Science and Technology（南京信息工程大学）；Autocity (Shenzhen) Autonomous Driving Co.,ltd（AutoCity（深圳）自动驾驶有限公司）；South China Normal University（华南师范大学）；Endeavor Star Co.,ltd（Endeavor Star有限公司）；Inspur Group（浪潮集团）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Peking University（北京大学）；University of New South Wales（新南威尔士大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Efficient ObjectGoal navigation (ObjectNav) in novel environments requires an understanding of the spatial and semantic regularities in environment layouts. In this work, we present a straightforward method for learning these regularities by predicting the locations of unobserved objects from incomplete semantic maps. Our method differs from previous prediction-based navigation methods, such as frontier potential prediction or egocentric map completion, by directly predicting unseen targets while leveraging the global context from all previously explored areas. Our prediction model is lightwei…
Robust three-dimensional scene understanding is now an ever-growing area of research highly relevant in many real-world applications such as autonomous driving and robotic navigation. In this paper, we propose a multi-task learning-based model capable of performing two tasks:- sparse depth completion (i.e. generating complete dense scene depth given a sparse depth image as the input) and monocular depth estimation (i.e. predicting scene depth from a single RGB image) via two sub-networks jointly trained end to end using data randomly sampled from a publicly available corpus of synthetic and re…
