This paper addresses the problem of efficiently achieving visual predictive control tasks. To this end, a memory of motion, containing a set of trajectories built off-line, is used for leveraging precomputation and dealing with difficult visual tasks. Standard regression techniques, such as k-nearest neighbors and Gaussian process regression, are used to query the memory and provide on-line a warm-start and a way point to the control optimization process. The proposed technique allows the control scheme to achieve high performance and, at the same time, keep the computational time limited. Simulation and experimental results, carried out with a 7-axis manipulator, show the effectiveness of the approach.
Tongji University（同济大学）；New York University（纽约大学）；Tsinghua University（清华大学）
As a crucial robotic perception capability, visual tracking has been intensively studied recently. In the real-world scenarios, the onboard processing time of the image streams inevitably leads to a discrepancy between the tracking results and the real-world states. However, existing visual tracking benchmarks commonly run the trackers offline and ignore such latency in the evaluation. In this work, we aim to deal with a more realistic problem of latency-aware tracking. The state-of-the-art trackers are evaluated in the aerial scenarios with new metrics jointly assessing the tracking accuracy …
Institute for Aerospace Studies, University of Toronto（多伦多大学航空航天研究所）
Navigation in unknown, chaotic environments continues to present a significant challenge for the robotics community. Lighting changes, self-similar textures, motion blur, and moving objects are all considerable stumbling blocks for state-of-the-art vision-based navigation algorithms. In this paper we present a novel technique for improving localization accuracy within a visual-inertial navigation system (VINS). We make use of training data to learn a model for the quality of visual features with respect to localization error in a given environment. This model maps each visual observation from …
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）
Mental simulation is a critical cognitive function for goal-directed behavior because it is essential for assessing actions and their consequences. When a self-generated or externally specified goal is given, a sequence of actions that is most likely to attain that goal is selected among other candidates via mental simulation. Therefore, better mental simulation leads to better goal-directed action planning. However, developing a mental simulation model is challenging because it requires knowledge of self and the environment. The current paper studies how adequate goal-directed action plans of…
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；The Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））
Learning to See through Action: Predictive Curiosity for Robots
HERON - Hellenic Robotics Center of Excellence, Athena Research Center（希腊雅典娜研究中心HERON希腊卓越机器人中心）；Institute of Robotics, Athena Research Center（雅典娜研究中心机器人研究所）；National Technical University of Athens, School of Electrical & Computer Engineering（雅典国立技术大学电气与计算机工程学院）
We present a perception constrained visual predictive control (PCVPC) algorithm for quadrotors to enable aggressive flights without using any position information. Our framework leverages nonlinear model predictive control (NMPC) to formulate a constrained image-based visual servoing (IBVS) problem. The quadrotor dynamics, image dynamics, actuation constraints, and visibility constraints are taken into account to handle quadrotor maneuvers with high agility. Two main challenges of applying IBVS to agile drones are considered: (i) high sensitivity of depths to intense orientation changes, and (…
Robotics Research Center, IIIT Hyderabad（海得拉巴国际信息技术学院机器人研究中心）；University Of Lincoln（林肯大学）；TCS Innovation Labs（塔塔咨询服务创新实验室）
The simplicity of the visual servoing approach makes it an attractive option for tasks dealing with vision-based control of robots in many real-world applications. However, attaining precise alignment for unseen environments pose a challenge to existing visual servoing approaches. While classical approaches assume a perfect world, the recent data-driven approaches face issues when generalizing to novel environments. In this paper, we aim to combine the best of both worlds. We present a deep model predictive visual servoing framework that can achieve precise alignment with optimal trajectories …
UC Berkeley（加州大学伯克利分校）；Facebook AI Research（Facebook人工智能研究院）；Technion（以色列理工学院）；OpenAI
We introduce a new memory architecture, Bayesian Relational Memory (BRM), to improve the generalization ability for semantic visual navigation agents in unseen environments, where an agent is given a semantic target to navigate towards. BRM takes the form of a probabilistic relation graph over semantic entities (e.g., room types), which allows (1) capturing the layout prior from training environments, i.e., prior knowledge, (2) estimating posterior layout at test time, i.e., memory update, and (3) efficient planning for navigation, altogether. We develop a BRM agent consisting of a BRM module …
Nara Institute of Science and Technology (NAIST)（奈良先端科学技术大学院大学）；Advanced Telecommunications Research Institute International (ATR)（国际高级电信研究所）；Kyoto University（京都大学）
Many works have recently explored Sim-to-real transferable visual model predictive control (MPC). However, such works are limited to one-shot transfer, where real-world data must be collected once to perform the sim-to-real transfer, which remains a significant human effort in transferring the models learned in simulations to new domains in the real world. To alleviate this problem, we first propose a novel model-learning framework called Kalman Randomized-to-Canonical Model (KRC-model). This framework is capable of extracting task-relevant intrinsic features and their dynamics from randomized…
Offline Deep Model Predictive Control (MPC) for Visual Navigation
International Artificial Intelligence Center of Morocco（摩洛哥国际人工智能中心）；Mohammed VI Polytechnic University（穆罕默德六世理工大学）；ENSAM（高等艺术与制造学院）；Moulay Ismail University（穆莱·伊斯梅尔大学）
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；Shanghai Jiao Tong University（上海交通大学）
This paper presents an image-based visual servo control (IBVS) method for a first-person-view (FPV) quadrotor to conduct aggressive aerial tracking. There are three major challenges to maneuvering an underactuated vehicle using IBVS: (i) finding a visual feature representation that is robust to large rotations and is suited to be an optimization variable; (ii) keeping the target visible without sacrificing the robot's agility; and (iii) compensating for the rotational effects in the detected features. We propose a complete design framework to address these problems. First, we employ a rotation…
University of California, Berkeley（加州大学伯克利分校）；New York University（纽约大学）
Using visual model-based learning for deformable object manipulation is challenging due to difficulties in learning plannable visual representations along with complex dynamic models. In this work, we propose a new learning framework that jointly optimizes both the visual representation model and the dynamics model using contrastive estimation. Using simulation data collected by randomly perturbing deformable objects on a table, we learn latent dynamics models for these objects in an offline fashion. Then, using the learned models, we use simple model-based planning to solve challenging deform…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Facebook AI Research（脸书人工智能研究院）；Technical University of Munich（慕尼黑工业大学）
Humans have impressive generalization capabilities when it comes to manipulating objects and tools in completely novel environments. These capabilities are, at least partially, a result of humans having internal models of their bodies and any grasped object. How to learn such body schemas for robots remains an open problem. In this work, we develop an self-supervised approach that can extend a robot's kinematic model when grasping an object from visual latent representations. Our framework comprises two components: (1) we present a multi-modal keypoint detector: an autoencoder architecture tra…
Predictive and adaptive maps for long-term visual navigation in changing environments
Artificial Intelligence Center, Czech Technical University（捷克技术大学人工智能中心）；Technical University of Cartagena, Spain（西班牙卡塔加纳技术大学）；Australian Centre for Robotic Vision, QUT（澳大利亚机器人视觉中心，昆士兰大学）
Luddy School of Informatics, Computing, and Engineering, Indiana University（印第安纳大学卢迪信息学、计算与工程学院）
In this paper, we open up new avenues for visual servoing systems built upon the Path Integral (PI) optimal control theory, in which the non-linear partial differential equation (PDE) can be transformed into an expectation over all possible trajectories using the Feynman-Kac (FK) lemma. More precisely, we propose an MPPI-VS control strategy, a real-time and inversion-free control strategy on the basis of sampling-based model predictive control (namely, Model Predictive Path Integral (MPPI) control) algorithm, for both image-based, 3D point, and position-based visual servoing techniques, taking…
Evolution Strategy (ES) algorithms have shown promising results in training complex robotic control policies due to their massive parallelism capability, simple implementation, effective parameter-space exploration, and fast training time. However, a key limitation of ES is its scalability to large capacity models, including modern neural network architectures. In this work, we develop Predictive Information Augmented Random Search (PI-ARS) to mitigate this limitation by leveraging recent advancements in representation learning to reduce the parameter search space for ES. Namely, PI-ARS combin…
