主动感知使机器人能够通过调整其视点来动态收集信息，这是与复杂、部分可观测环境交互的一项关键能力。在本文中，我们提出了 AP-VLM，这是一种将主动感知与视觉-语言模型（VLM）相结合的新颖框架，用于指导机器人探索并回答语义查询。通过使用叠加在场景上的三维虚拟网格以及朝向调整，AP-VLM 使机器人机械臂能够智能地选择最佳视点和朝向，以解决具有挑战性的任务，例如识别处于被遮挡或倾斜位置的物体。我们在两个机器人平台上评估了我们的系统：一台 7-DOF Franka Panda 和一台 6-DOF UR5，场景涵盖具有不同物体配置的多种场景。我们的结果表明，AP-VLM 显著优于被动感知方法和基线模型，包括 Toward Grounded Common Sense Reasoning（TGCSR），尤其是在固定相机视图不足的场景中。AP-VLM 在真实世界环境中的适应性展现出增强机器人系统理解复杂环境的前景，弥合了高层语义推理与低层控制之间的差距。
英文摘要：
Active perception enables robots to dynamically gather information by adjusting their viewpoints, a crucial capability for interacting with complex, partially observable environments. In this paper, we present AP-VLM, a novel framework that combines active perception with a Vision-Language Model (VLM) to guide robotic exploration and answer semantic queries. Using a 3D virtual grid overlaid on the scene and orientation adjustments, AP-VLM allows a robotic manipulator to intelligently select optimal viewpoints and orientations to resolve challenging tasks, such as identifying objects in occluded or inclined positions. We evaluate our system on two robotic platforms: a 7-DOF Franka Panda and a 6-DOF UR5, across various scenes with differing object configurations. Our results demonstrate that AP-VLM significantly outperforms passive perception methods and baseline models, including Toward Grounded Common Sense Reasoning (TGCSR), particularly in scenarios where fixed camera views are inadequate. The adaptability of AP-VLM in real-world settings shows promise for enhancing robotic systems' understanding of complex environments, bridging the gap between high-level semantic reasoning and low-level control.
Contact SLAM: An Active Tactile Exploration Policy Based on Physical Reasoning Utilized in Robotic Fine Blind Manipulation Tasks
Research Center for Intelligent Robotics（智能机器人研究中心）；National Key Laboratory of Aerospace Flight Dynamics（航空飞行动力学国家重点实验室）；Shaanxi Province Innovation Team of Intelligent Robotic Technology（陕西省智能机器人技术创新团队）；School of Astronautics, Northwestern Polytechnical University（航空学院，西北工业大学）
We introduce a novel method to teach a robotic agent to interactively explore cluttered yet structured scenes, such as kitchen pantries and grocery shelves, by leveraging the physical plausibility of the scene. We propose a novel learning framework to train an effective scene exploration policy to discover hidden objects with minimal interactions. First, we define a novel scene grammar to represent structured clutter. Then we train a Graph Neural Network (GNN) based Scene Generation agent using deep reinforcement learning (deep RL), to manipulate this Scene Grammar to create a diverse set of s…
Integrated Exploration and Sequential Manipulation on Scene Graph with LLM-based Situated Replanning
Beihang University（北航大学）；State Key Laboratory of General Artificial Intelligence（一般人工智能国家重点实验室）；University of California, Los Angeles（加州大学洛杉矶分校）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Applied Ocean Physics and Engineering Department at the Woods Hole Oceanographic Institution (WHOI)（伍兹霍尔海洋研究所（WHOI）应用海洋物理与工程系）；MIT Media Lab（麻省理工学院媒体实验室）；MIT-WHOI Joint Program in Applied Ocean Science and Engineering（麻省理工学院-伍兹霍尔海洋研究所应用海洋科学与工程联合项目）；Mechanical Engineering Department at Northeastern University（东北大学机械工程系）
This paper proposes a bandwidth tunable technique for real-time probabilistic scene modeling and mapping to enable co-robotic exploration in communication constrained environments such as the deep sea. The parameters of the system enable the user to characterize the scene complexity represented by the map, which in turn determines the bandwidth requirements. The approach is demonstrated using an underwater robot that learns an unsupervised scene model of the environment and then uses this scene model to communicate the spatial distribution of various high-level semantic scene constructs to a h…
Australian Research Council Centre of Excellence for Robotic Vision（澳大利亚研究理事会机器人视觉卓越中心）；QUT Centre for Robotics（昆士兰科技大学机器人中心）
Being able to explore an environment and understand the location and type of all objects therein is important for indoor robotic platforms that must interact closely with humans. However, it is difficult to evaluate progress in this area due to a lack of standardized testing which is limited due to the need for active robot agency and perfect object ground-truth. To help provide a standard for testing scene understanding systems, we present a new robot vision scene understanding challenge using simulation to enable repeatable experiments with active robot agency. We provide two challenging tas…
We address the problem of autonomous exploration and mapping for a mobile robot using visual inputs. Exploration and mapping is a well-known and key problem in robotics, the goal of which is to enable a robot to explore a new environment autonomously and create a map for future usage. Different to classical methods, we propose a learning-based approach this work based on semantic interpretation of visual scenes. Our method is based on a deep network consisting of three modules: semantic segmentation network, mapping using camera geometry and exploration action network. All modules are differen…
Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control
Chair of Robotics and Systems Intelligence, MIRMI-Munich Institute of Robotics and Machine Intelligence, Technical University of Munich（慕尼黑工业大学机器人与系统智能教席，慕尼黑机器人与机器智能研究所）；Centre for Tactile Internet with Human-in-the-Loop (CeTI)（以人为中心的触觉互联网中心）
ETH Zürich（苏黎世联邦理工学院）；Technical University of Munich（慕尼黑工业大学）；Google（谷歌）
Exploration of unknown environments is a fundamental problem in robotics and an essential component in numerous applications of autonomous systems. A major challenge in exploring unknown environments is that the robot has to plan with the limited information available at each time step. While most current approaches rely on heuristics and assumption to plan paths based on these partial observations, we instead propose a novel way to integrate deep learning into exploration by leveraging 3D scene completion for informed, safe, and interpretable exploration mapping and planning. Our approach, SC…
ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models
State Key Laboratory of Industrial Control Technology（工业控制技术国家重点实验室）；Institute of Cyber-Systems and Control（网络系统与控制研究所）；Zhejiang University（浙江大学）；Zhejiang Humanoid Robot Innovation Center Co., Ltd.（浙江人形机器人创新中心有限公司）
RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Columbia University（哥伦比亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon（亚马逊公司）
We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial and geometric reasoning skills on top of those abstractions in…
City University of Hong Kong（香港城市大学）；Hong Kong University of Science and Technology（香港科技大学）
Exploration in an unknown environment is the core functionality for mobile robots. Learning-based exploration methods, including convolutional neural networks, provide excellent strategies without human-designed logic for the feature extraction. But the conventional supervised learning algorithms cost lots of efforts on the labeling work of datasets inevitably. Scenes not included in the training set are mostly unrecognized either. We propose a deep reinforcement learning method for the exploration of mobile robots in an indoor environment with the depth information from an RGB-D sensor only. …
Insects have tiny brains but complicated visual systems for motion perception. A handful of insect visual neurons have been computationally modeled and successfully applied for robotics. How different neurons collaborate on motion perception, is an open question to date. In this paper, we propose a novel embedded vision system in autonomous micro-robots, to recognize motion patterns in dynamic robot scenes. Here, the basic motion patterns are categorized into movements of looming (proximity), recession, translation, and other irrelevant ones. The presented system is a synthetic neural network,…
Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving
ECE Department（电子工程系）；Carnegie Mellon University（卡内基梅隆大学）；Computer Science Department（计算机科学系）；Columbia University（哥伦比亚大学）；Rotman School of Management（罗特曼管理学院）；University of Toronto（多伦多大学）；Department of Statistics（统计学系）；George Washington University（乔治华盛顿大学）；Department of Computer Science（计算机科学系）；San Francisco State University（旧金山州立大学）
