Attention control is a key cognitive ability for humans to select information relevant to the current task. This paper develops a computational model of attention and an algorithm for attention-based probabilistic planning in Markov decision processes. In attention-based planning, the robot decides to be in different attention modes. An attention mode corresponds to a subset of state variables monitored by the robot. By switching between different attention modes, the robot actively perceives task-relevant information to reduce the cost of information acquisition and processing, while achieving near-optimal task performance. Though planning with attention-based active perception inevitably introduces partial observations, a partially observable MDP formulation makes the problem computational expensive to solve. Instead, our proposed method employs a hierarchical planning framework in which the robot determines what to pay attention to and for how long the attention should be sustained before shifting to other information sources. During the attention sustaining phase, the robot carries out a sub-policy, computed from an abstraction of the original MDP given the current attention. We use an example where a robot is tasked to capture a set of intruders in a stochastic gridworld. The experimental results show that the proposed method enables information- and computation-efficient optimal planning in stochastic environments.
Virginia Tech（弗吉尼亚理工大学）；University of Maryland（马里兰大学）
We propose a risk-aware framework for multi-robot, multi-demand assignment and planning in unknown environments. Our motivation is disaster response and search-and-rescue scenarios where ground vehicles must reach demand locations as soon as possible. We consider a setting where the terrain information is available only in the form of an aerial, georeferenced image. Deep learning techniques can be used for semantic segmentation of the aerial image to create a cost map for safe ground robot navigation. Such segmentation may still be noisy. Hence, we present a joint planning and perception frame…
University of California at Berkeley（加州大学伯克利分校）；California PATH, University of California at Berkeley（加州大学伯克利分校加州PATH）
Autonomous cars have to navigate in dynamic environment which can be full of uncertainties. The uncertainties can come either from sensor limitations such as occlusions and limited sensor range, or from probabilistic prediction of other road participants, or from unknown social behavior in a new area. To safely and efficiently drive in the presence of these uncertainties, the decision-making and planning modules of autonomous cars should intelligently utilize all available information and appropriately tackle the uncertainties so that proper driving strategies can be generated. In this paper, …
Most of the works on planning and learning, e.g., planning by (model based) reinforcement learning, are based on two main assumptions: (i) the set of states of the planning domain is fixed; (ii) the mapping between the observations from the real word and the states is implicitly assumed or learned offline, and it is not part of the planning domain. Consequently, the focus is on learning the transitions between states. In this paper, we drop such assumptions. We provide a formal framework in which (i) the agent can learn dynamically new states of the planning domain; (ii) the mapping between ab…
FLAP: FOV-Constrained Active Perception Planning for Prior-Map-Free 3D Navigation
The State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院工业控制技术国家重点实验室）；Huzhou Institute, Zhejiang University（浙江大学湖州研究院）；Huzhou Key Laboratory of Autonomous System（湖州市自动驾驶系统重点实验室）；Shanghai Institute of Special Equipment Inspection and Technical Research Co., Ltd（上海市特种设备监督检验技术研究院有限公司）
Universidade Federal do Rio Grande (FURG)（里奥格兰德联邦大学（FURG））；Universidad Tecnológica del Uruguay (UTEC)（乌拉圭技术大学（UTEC））
Unmanned Aerial Vehicles (UAV) have been standing out due to the wide range of applications in which they can be used autonomously. However, they need intelligent systems capable of providing a greater understanding of what they perceive to perform several tasks. They become more challenging in complex environments since there is a need to perceive the environment and act under environmental uncertainties to make a decision. In this context, a system that uses active perception can improve performance by seeking the best next view through the recognition of targets while displacement occurs. T…
Active perception has been employed in many domains, particularly in the field of robotics. The idea of active perception is to utilize the input data to predict the next action that can help robots to improve their performance. The main challenge lies in understanding the input data to be coupled with the action, and gathering meaningful information of the environment in an efficient way is necessary and desired. With recent developments of neural networks, interpreting the perceived data has become possible at the semantic level, and real-time interpretation based on deep learning has enable…
Siemens（西门子）；California Institute of Technology（加州理工学院）
Active perception systems maximizing information gain to support both monitoring and decision making have seen considerable application in recent work. In this paper, we propose and demonstrate a method of acquiring and extrapolating information in an active sensory system through use of a Bayesian Filter. Our approach is motivated by manufacturing processes, where automated visual tracking of system states may aid in fault diagnosis, certification of parts and safety; in extreme cases, our approach may enable novel manufacturing processes relying on monitoring solutions beyond passive percept…
Apple: Toward General Active Perception via Reinforcement Learning
Department of Computer Science, TU Darmstadt, Germany（德国图林根大学计算机科学系）；LIRIS, CNRS UMR5205, École Centrale de Lyon, France（法国里里萨大学LIRIS实验室）；LASR Lab & CeTI, TU Dresden, Germany（德国德累斯顿技术大学LASR实验室）；DFKI, Hessian.AI, RIG, and Centre for Cognitive Science, TU Darmstadt, Germany（德国图林根大学DFKI、海德堡人工智能、RIG及认知科学中心）
Active perception strategies enable an agent to selectively gather information in a way to improve its performance. In applications in which the agent does not have prior knowledge about the available information sources, it is crucial to synthesize active perception strategies at runtime. We consider a setting in which at runtime an agent is capable of gathering information under a limited budget. We pose the problem in the context of partially observable Markov decision processes. We propose a generalized greedy strategy that selects a subset of information sources with near-optimality guara…
University of Maryland（马里兰大学）；University of California, Berkeley（加州大学伯克利分校）；York University（约克大学）
Despite the recent successes in robotics, artificial intelligence and computer vision, a complete artificial agent necessarily must include active perception. A multitude of ideas and methods for how to accomplish this have already appeared in the past, their broader utility perhaps impeded by insufficient computational power or costly hardware. The history of these ideas, perhaps selective due to our perspectives, is presented with the goal of organizing the past literature and highlighting the seminal contributions. We argue that those contributions are as relevant today as they were decades…
University of Amsterdam（阿姆斯特丹大学）；University of Oxford（牛津大学）；University of Liverpool（利物浦大学）；Delft University of Technology（代尔夫特理工大学）
In active perception tasks, an agent aims to select sensory actions that reduce its uncertainty about one or more hidden variables. While partially observable Markov decision processes (POMDPs) provide a natural model for such problems, reward functions that directly penalize uncertainty in the agent's belief can remove the piecewise-linear and convex property of the value function required by most POMDP planners. Furthermore, as the number of sensors available to the agent grows, the computational cost of POMDP planning grows exponentially with it, making POMDP planning infeasible with tradit…
The University of Adelaide（阿德莱德大学）；InnoPeak Technology, Inc.（InnoPeak科技公司）；Monash University（莫纳什大学）；Australian National University（澳大利亚国立大学）
A high-quality 3D reconstruction of a scene from a collection of 2D images can be achieved through offline/online mapping methods. In this paper, we explore active mapping from the perspective of implicit representations, which have recently produced compelling results in a variety of applications. One of the most popular implicit representations - Neural Radiance Field (NeRF), first demonstrated photorealistic rendering results using multi-layer perceptrons, with promising offline 3D reconstruction as a by-product of the radiance field. More recently, researchers also applied this implicit re…
ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大模型与智能治理重点实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Jiaotong University（北京交通大学）；State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；AresoX
Most real-world 3D sensors such as LiDARs perform fixed scans of the entire environment, while being decoupled from the recognition system that processes the sensor data. In this work, we propose a method for 3D object recognition using light curtains, a resource-efficient controllable sensor that measures depth at user-specified locations in the environment. Crucially, we propose using prediction uncertainty of a deep learning based 3D point cloud detector to guide active perception. Given a neural network's uncertainty, we derive an optimization objective to place light curtains using the pr…
Robotics Research Center, IIIT Hyderabad（国际信息技术研究所海得拉巴分校机器人研究中心）；TCS Research（塔塔咨询服务公司研究部）
Table-top Rearrangement and Planning is a challenging problem that relies heavily on an excellent perception stack. The perception stack involves observing and registering the 3D scene on the table, detecting what objects are on the table, and how to manipulate them. Consequently, it greatly influences the system's task-planning and motion-planning stacks that follow. We present a comprehensive overview and discuss the different challenges associated with the perception module. This work is a result of our extensive involvement in the ICRA 2022 Open Cloud Robot Table Organization Challenge, in…
In this paper we describe a framework towards computing well-localized, robust motion plans through the perception-aware motion planning problem, whereby we seek a low-cost motion plan subject to a separate constraint on perception localization quality. To solve this problem we introduce the Multiobjective Perception-Aware Planning (MPAP) algorithm which explores the state space via a multiobjective search, considering both cost and a perception heuristic. This framework can accommodate a large range of heuristics, allowing those that capture the history dependence of localization drift and re…
AppleVLM: End-to-end Autonomous Driving with Advanced Perception and Planning-Enhanced Vision-Language Models
Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments（深圳先进运动控制系统与现代自动化装备重点实验室）；Guangdong Provincial Key Laboratory of Intelligent Morphing Mechanisms and Adaptive Robotics（广东省智能变形机制与适应机器人省重点实验室）；School of Intelligence Science and Engineering（智能科学与工程学院）；Harbin Institute of Technology（哈尔滨工业大学）；National Key Laboratory of Smart Farm Technologies and Systems（国家智能农业技术与系统重点实验室）；Autonomous Driving Center（自动驾驶中心）；Shanghai Utopilot Technology Co.Ltd.（上海驭目科技有限公司）
This paper addresses a multi-robot planning problem in environments with partially unknown semantics. The environment is assumed to have known geometric structure (e.g., walls) and to be occupied by static labeled landmarks with uncertain positions and classes. This modeling approach gives rise to an uncertain semantic map generated by semantic SLAM algorithms. Our goal is to design control policies for robots equipped with noisy perception systems so that they can accomplish collaborative tasks captured by global temporal logic specifications. To specify missions that account for environmenta…
Lehigh University（利哈伊大学）；Eindhoven University of Technology（埃因霍温理工大学）
This work presents a step towards utilizing incrementally-improving symbolic perception knowledge of the robot's surroundings for provably correct reactive control synthesis applied to an autonomous driving problem. Combining abstract models of motion control and information gathering, we show that assume-guarantee specifications (a subclass of Linear Temporal Logic) can be used to define and resolve traffic rules for cautious planning. We propose a novel representation called symbolic refinement tree for perception that captures the incremental knowledge about the environment and embodies the…
Most existing motion planning algorithms assume that a map (of some quality) is fully determined prior to generating a motion plan. In many emerging applications of robotics, e.g., fast-moving agile aerial robots with constrained embedded computational platforms and visual sensors, dense maps of the world are not immediately available, and they are computationally expensive to construct. We propose a new algorithm for generating plan graphs which couples the perception and motion planning processes for computational efficiency. In a nutshell, the proposed algorithm iteratively switches between…
Look as You Leap: Planning Simultaneous Motion and Perception for High-DOF Robots
Department of Computer Science, Rice University（计算机科学系，里士大学）；Ken Kennedy Institute, Rice University（肯尼迪研究所，里士大学）；Houston Methodist Academic Institute（休斯顿 Methodist 学术研究院）
Istanbul Technical University（伊斯坦布尔理工大学）；Turkish-German University（土耳其-德国大学）
We investigate how to utilize predictive models for selecting appropriate motion planning strategies based on perception uncertainty estimation for agile unmanned aerial vehicle (UAV) navigation tasks. Although there are variety of motion planning and perception algorithms for such tasks, the impact of perception uncertainty is not explicitly handled in many of the current motion algorithms, which leads to performance loss in real-life scenarios where the measurement are often noisy due to external disturbances. We develop a novel framework for embedding perception uncertainty to high level mo…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Alliance Innovation Lab Silicon Valley（硅谷联盟创新实验室）
Robots deployed in the real world over extended periods of time need to reason about unexpected failures, learn to predict them, and to proactively take actions to avoid future failures. Existing approaches for competence-aware planning are either model-based, requiring explicit enumeration of known failure modes, or purely statistical, using state- and location-specific failure statistics to infer competence. We instead propose a structured model-free approach to competence-aware planning by reasoning about plan execution failures due to errors in perception, without requiring a priori enumer…
