Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, from physical properties such as friction and mass to human-defined environmental rules. We introduce T2-Z for test-time optimization of environment latents, which augments a robotic world model with a learnable environment latent to condition world predictions. This compact latent captures underlying environmental variation without supervision of what environmental factors exist or what values they take. During training, T2-Z alternates between two stages: a world-model optimization stage that fixes the latents and learns general action-conditioned dynamics across environments; and an environment-latent optimization stage that freezes the world model and optimizes the conditioning latents, allowing latent structure to emerge purely through world prediction. At test time, the world model remains frozen and a newly initialized environment latent is optimized from one or more test-time interactions, adapting the model to previously unseen environmental configurations. Across 7 simulated settings and 4 real-world robotic tasks, T2-Z reduces object prediction error metric from 15.20 to 7.00 and raises action success from 33.73% to 72.96%. Its emergent latents capture structured representations of individual and compounded factors across diverse visual scenes. Ablations further show that T2-Z can efficiently infer environment latents from as little as a single test-time interaction trajectory.
Few-Shot Adaptation to Non-Stationary Environments via Latent Trend Embedding for Robotics
College of Information Science and Engineering, Ritsumeikan University（立命馆大学信息科学与工程学院）；Production & Technology Department, NIPPN CORPORATION（NIPPN公司生产与技术部门）；University of Osaka（大阪大学）；National Institute of Technology, Ichinoseki College（国立Ichinoseki学院）
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；St. Mark’s School of Texas（德克萨斯圣马克学校）
We introduce a neural implicit representation for grasps of objects from multiple robotic hands. Different grasps across multiple robotic hands are encoded into a shared latent space. Each latent vector is learned to decode to the 3D shape of an object and the 3D shape of a robotic hand in a grasping pose in terms of the signed distance functions of the two 3D shapes. In addition, the distance metric in the latent space is learned to preserve the similarity between grasps across different robotic hands, where the similarity of grasps is defined according to contact regions of the robotic hands…
SurgLAT: Surgical Latent Attention Tracking for Depth-Aware Robotic Laparoscope Control
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Loop Area Institute（深圳河套学院）；Shenzhen University（深圳大学）；Southern University of Science and Technology（南方科技大学）
University of Hamburg（汉堡大学）；Knowledge Technology, Department of Informatics（知识技术与信息学系）
Combining model-based and model-free deep reinforcement learning has shown great promise for improving sample efficiency on complex control tasks while still retaining high performance. Incorporating imagination is a recent effort in this direction inspired by human mental simulation of motor behavior. We propose a learning-adaptive imagination approach which, unlike previous approaches, takes into account the reliability of the learned dynamics model used for imagining the future. Our approach learns an ensemble of disjoint local dynamics models in latent space and derives an intrinsic reward…
Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；X-Humanoid Robots（X-Humanoid机器人）；The University of Hong Kong（香港大学）；Tsinghua University（清华大学）
Chain of World: World Model Thinking in Latent Motion
Harbin Institute of Technology（哈尔滨工业大学）；Li Auto；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；University of New South Wales（新南威尔士大学）；Chongqing Research Institute of HIT（哈尔滨工业大学重庆研究院）；Peking University（北京大学）
Modeling the world can benefit robot learning by providing a rich training signal for shaping an agent's latent state space. However, learning world models in unconstrained environments over high-dimensional observation spaces such as images is challenging. One source of difficulty is the presence of irrelevant but hard-to-model background distractions, and unimportant visual details of task-relevant entities. We address this issue by learning a recurrent latent dynamics model which contrastively predicts the next observation. This simple model leads to surprisingly robust robotic control even…
Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Tasks
Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；AMI Labs（AMI实验室）
World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；Beijing Zhongke Huiling Robot Technology Co（北京中科创联机器人技术有限公司）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
INESC-ID（里斯本系统与计算机工程研究所）；Instituto Superior Técnico, University of Lisbon（里斯本大学高等技术学院）
We present Perceive-Represent-Generate (PRG), a novel three-stage framework that maps perceptual information of different modalities (e.g., visual or sound), corresponding to a sequence of instructions, to an adequate sequence of movements to be executed by a robot. In the first stage, we perceive and pre-process the given inputs, isolating individual commands from the complete instruction provided by a human user. In the second stage we encode the individual commands into a multimodal latent space, employing a deep generative model. Finally, in the third stage we convert the multimodal latent…
Robotic agents that operate autonomously in the real world need to continuously explore their environment and learn from the data collected, with minimal human supervision. While it is possible to build agents that can learn in such a manner without supervision, current methods struggle to scale to the real world. Thus, we propose ALAN, an autonomously exploring robotic agent, that can perform tasks in the real world with little training and interaction time. This is enabled by measuring environment change, which reflects object movement and ignores changes in the robot position. We use this m…
GWM: Towards Scalable Gaussian World Models for Robotic Manipulation
Tsinghua University（清华大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）
Human-robot interaction often occurs in the form of instructions given from a human to a robot. For a robot to successfully follow instructions, a common representation of the world and objects in it should be shared between humans and the robot so that the instructions can be grounded. Achieving this representation can be done via learning, where both the world representation and the language grounding are learned simultaneously. However, in robotics this can be a difficult task due to the cost and scarcity of data. In this paper, we tackle the problem by separately learning the world represe…
University of Glasgow（格拉斯哥大学）；University of Verona（维罗纳大学）
We present an introspective framework inspired by the process of how humans perform introspection. Our working assumption is that neural network activations encode information, and building internal states from these activations can improve the performance of an actor-critic model. We perform experiments where we first train a Variational Autoencoder model to reconstruct the activations of a feature extraction network and use the latent space to improve the performance of an actor-critic when deciding which low-level robotic behaviour to execute. We show that internal states reduce the number …
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Tsinghua University（清华大学）；Jilin University（吉林大学）；Nankai University（南开大学）；Peking University（北京大学）；Harbin Institute of Technology（哈尔滨工业大学）；Zhongguancun Academy（中关村学院）；Striding.AI；Infinigence AI
Ritsumeikan University（立命馆大学）；Keio University（庆应义塾大学）；The University of Tokyo（东京大学）；Università Milano-Bicocca（米兰比可卡大学）；University of Essex（埃塞克斯大学）；Donders Institute for Brain, Cognition and Behaviour（唐德斯脑、认知与行为研究所）；Cajal International Neuroscience Center, Spanish National Research Council（西班牙国家研究委员会卡哈尔国际神经科学中心）；Bogazici University（博阿齐奇大学）；Queen Mary University of London（伦敦玛丽女王大学）；The University of Electro-Communications（电气通信大学）；National Autonomous University of Mexico（墨西哥国立自治大学）；Universidad Autónoma del Estado de Morelos（莫雷洛斯州自治大学）；Institute of Cognitive Sciences and Technologies, National Research Council of Italy（意大利国家研究委员会认知科学与技术研究所）
Creating autonomous robots that can actively explore the environment, acquire knowledge and learn skills continuously is the ultimate achievement envisioned in cognitive and developmental robotics. Their learning processes should be based on interactions with their physical and social world in the manner of human learning and cognitive development. Based on this context, in this paper, we focus on the two concepts of world models and predictive coding. Recently, world models have attracted renewed attention as a topic of considerable interest in artificial intelligence. Cognitive systems learn…
A Retrospective on the Robot Air Hockey Challenge: Benchmarking Robust, Reliable, and Safe Learning Techniques for Real-world Robotics
Intelligent Autonomous Systems, TU Darmstadt（达姆施塔特工业大学智能自主系统研究所）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Huawei German research center（华为德国研究中心）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University College London（伦敦大学学院）；Idiap Research Institute（伊迪亚普研究所）；EPFL（洛桑联邦理工学院）；University of Luxembourg（卢森堡大学）；Karlsruher Institut für Technologie（卡尔斯鲁厄理工学院）；Universita di Bologna（博洛尼亚大学）；Politecnico di Milano（米兰理工大学）；Centre for Cognitive Science（认知科学中心）
University of Hamburg（汉堡大学）；Knowledge Technology（知识技术学院）
Combining model-based and model-free learning systems has been shown to improve the sample efficiency of learning to perform complex robotic tasks. However, dual-system approaches fail to consider the reliability of the learned model when it is applied to make multiple-step predictions, resulting in a compounding of prediction errors and performance degradation. In this paper, we present a novel dual-system motor learning approach where a meta-controller arbitrates online between model-based and model-free decisions based on an estimate of the local reliability of the learned model. The reliab…
Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world robotic tasks. In this paper, we study how maximum entropy policies trained using soft Q-learning can be applied to real-world robotic manipulation. The application of this method to real-world manipulation is facilitated by two important features of soft Q-learning. First, soft Q-…
