Dynamic Neural Koopman Distillation for Fast Robot Control Using Diffusion Models
Department of Electrical and Computer Engineering, National University of Singapore（国立新加坡大学电子与计算机工程系）；Department of Electrical and Computer Engineering, Carnegie Mellon University（卡内基梅隆大学电子与计算机工程系）；Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology（香港科学与技术大学机器人与自主系统方向）
Lei Zheng, Peiqi Yu, Zengqi Peng, Changliu Liu, Armin Lederer
Diffusion models excel at generating diverse, multimodal trajectories for robotic control, yet their iterative denoising process introduces latency that can constrain real-time closed-loop execution. To address this problem, we propose a Dynamic Neural Koopman (DNK) distillation framework, which distills multistep diffusion inference into a single forward pass conditioned on observations and sampled noise. Specifically, we introduce a Factorized Dynamic Koopman (FDK) layer that models the denoising process through a latent linear transition. The noisy trajectory input is lifted into a latent space, where the FDK layer parameterizes a factorized linear operator with condition-dependent eigenvalues that adapt the noisy-to-denoised transition to the current context. We evaluate DNK across robot-control benchmarks that span locomotion, state-based, and image-based manipulation, comparing against accelerated generative-policy baselines. The results demonstrate that DNK achieves competitive or improved closed-loop performance relative to accelerated generative-policy baselines, while substantially reducing inference latency relative to the strongest high-performing one-step baselines. On long-horizon manipulation tasks, DNK achieves competitive performance with substantially fewer inference parameters than the one-step policy baselines. Hardware experiments on a physical Kinova manipulator further demonstrate millisecond-scale inference and reduced task completion time. A project page is available at https://fdkoopman.github.io/.
Purpose of review: We review recent advances in algorithmic development and validation for modeling and control of soft robots leveraging the Koopman operator theory. Recent findings: We identify the following trends in recent research efforts in this area. (1) The design of lifting functions used in the data-driven approximation of the Koopman operator is critical for soft robots. (2) Robustness considerations are emphasized. Works are proposed to reduce the effect of uncertainty and noise during the process of modeling and control. (3) The Koopman operator has been embedded into different mo…
Controlling soft robots with precision is a challenge due in large part to the difficulty of constructing models that are amenable to model-based control design techniques. Koopman Operator Theory offers a way to construct explicit linear dynamical models of soft robots and to control them using established model-based linear control methods. This method is data-driven, yet unlike other data-driven models such as neural networks, it yields an explicit control-oriented linear model rather than just a "black-box" input-output mapping. This work describes this Koopman-based system identification …
Soft robots are challenging to model and control as inherent non-linearities (e.g., elasticity and deformation), often requires complex explicit physics-based analytical modeling (e.g., a priori geometric definitions). While machine learning can be used to learn non-linear control models in a data-driven approach, these models often lack an intuitive internal physical interpretation and representation, limiting dynamical analysis. To address this, this paper presents an approach using Koopman operator theory and deep neural networks to provide a global linear description of the non-linear cont…
Shape control of simulated multi-segment continuum robots via Koopman operators with per-segment projection
Department of Electrical Engineering and Computer Science University of Michigan, Ann Arbor, Michigan 48109, USA（密歇根大学电气工程与计算机科学系）；School for Engineering of Matter, Transport and Energy Arizona State University, Tempe, Arizona 85281, USA（亚利桑那州立大学工程物质、运输与能源学院）
This paper presents an active learning strategy for robotic systems that takes into account task information, enables fast learning, and allows control to be readily synthesized by taking advantage of the Koopman operator representation. We first motivate the use of representing nonlinear systems as linear Koopman operator systems by illustrating the improved model-based control performance with an actuated Van der Pol system. Information-theoretic methods are then applied to the Koopman operator formulation of dynamical systems where we derive a controller for active learning of robot dynamic…
Modeling and Control of a Pneumatic Soft Robotic Catheter Using Neural Koopman Operators
Laboratory for Computational Sensing and Robotics（计算感知与机器人实验室）；Johns Hopkins University（约翰霍普金斯大学）；Department of Mechanical Engineering（机械工程系）；University of Maryland（马里兰大学）
Soft robots are challenging to model due in large part to the nonlinear properties of soft materials. Fortunately, this softness makes it possible to safely observe their behavior under random control inputs, making them amenable to large-scale data collection and system identification. This paper implements and evaluates a system identification method based on Koopman operator theory in which models of nonlinear dynamical systems are constructed via linear regression of observed data by exploiting the fact that every nonlinear system has a linear representation in the infinite-dimensional spa…
Koopman operator theory has served as the basis to extract dynamics for nonlinear system modeling and control across settings, including non-holonomic mobile robot control. There is a growing interest in research to derive robustness (and/or safety) guarantees for systems the dynamics of which are extracted via the Koopman operator. In this paper, we propose a way to quantify the prediction error because of noisy measurements when the Koopman operator is approximated via Extended Dynamic Mode Decomposition. We further develop an enhanced robot control strategy to endow robustness to a class of…
Technische Universität Ilmenau（伊尔梅瑙工业大学）；University of Stuttgart（斯图加特大学）
Data-driven surrogate models of dynamical systems based on the extended dynamic mode decomposition are nowadays well-established and widespread in applications. Further, for non-holonomic systems exhibiting a multiplicative coupling between states and controls, the usage of bi-linear surrogate models has proven beneficial. However, an in-depth analysis of the approximation quality and its dependence on different hyperparameters based on both simulation and experimental data is still missing. We investigate a differential-drive mobile robot to close this gap and provide first guidelines on the …
SKooP: Symmetric Koopman Predictions for Faster and More Generalizable Legged Robot Locomotion with Reinforcement Learning
Italian Institute of Technology (IIT)（意大利理工学院）；University of Manchester（曼彻斯特大学）；Dynamic Legged Systems Laboratory, IIT（意大利理工学院动态腿式系统实验室）；Generative Bionics S.R.L（生成仿生学有限公司）；DAUIN, Politecnico di Torino（都灵理工大学DAUIN）；Rehab Technologies Lab, IIT（意大利理工学院康复技术实验室）；Newcastle University（纽卡斯尔大学）
Dept. of Electrical and Computer Engineering, University of California, Riverside（加州大学河滨分校电气与计算机工程系）；Dept. of Mechanical and Aerospace Engineering, University of California San Diego（加州大学圣地亚哥分校机械与航空航天工程系）；Dept. of Planning and Control, Zoox Inc.（Zoox公司规划与控制部）；Dept. of Mechanical Engineering, University of Michigan（密歇根大学机械工程系）；Dept. of Mechanical Engineering, Yale University（耶鲁大学机械工程系）；Dept. of Mechanical Engineering, Northwestern University（西北大学机械工程系）；Institute of AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
This paper explores the application of Koopman operator theory to the control of robotic systems. The operator is introduced as a method to generate data-driven models that have utility for model-based control methods. We then motivate the use of the Koopman operator towards augmenting model-based control. Specifically, we illustrate how the operator can be used to obtain a linearizable data-driven model for an unknown dynamical process that is useful for model-based control synthesis. Simulated results show that with increasing complexity in the choice of the basis functions, a closed-loop co…
Koopman-based learning methods can potentially be practical and powerful tools for dynamical robotic systems. However, common methods to construct Koopman representations seek to learn lifted linear models that cannot capture nonlinear actuation effects inherent in many robotic systems. This paper presents a learning and control methodology that is a first step towards overcoming this limitation. Using the Koopman canonical transform, control-affine dynamics can be expressed by a lifted bilinear model. The learned model is used for nonlinear model predictive control (NMPC) design where the bil…
Soft grippers are gaining momentum across applications due to their flexibility and dexterity. However, the infinite-dimensionality and non-linearity associated with soft robots challenge modeling and closed-loop control of soft grippers to perform grasping tasks. To solve this problem, data-driven methods have been proposed. Most data-driven methods rely on intensive model learning in simulation or offline, and as such it may be hard to generalize across different settings not explicitly trained upon and in physical robot testing where online control is required. In this paper, we propose an …
A Koopman Operator-based NMPC Framework for Mobile Robot Navigation under Uncertainty
Department of Electrical and Computer Engineering, University of California, Riverside（加州大学河滨分校电气与计算机工程系）；Polytechnic School of Tunisia（突尼斯多技术学院）；Institute of AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Soft robots promise improved safety and capability over rigid robots when deployed in complex, delicate, and dynamic environments. However, the infinite degrees of freedom and highly nonlinear dynamics of these systems severely complicate their modeling and control. As a step toward addressing this open challenge, we apply the data-driven, Hankel Dynamic Mode Decomposition (HDMD) with time delay observables to the model identification of a highly inertial, helical soft robotic arm with a high number of underactuated degrees of freedom. The resulting model is linear and hence amenable to contro…
Multi-segment Soft Robot Control via Deep Koopman-based Model Predictive Control
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；School of Biomedical Engineering, Tsinghua University（清华大学生物医学工程学院）；Beijing Soft Robot Tech Co., Ltd.（北京软机器人技术有限公司）；School of Clinical Medicine, Tsinghua University（清华大学临床医学学院）；Department of Informatics, University of Hamburg（汉堡大学信息学院）；School of Mechanical Engineering and Automation, Beihang University（北航机械工程与自动化学院）；Huawei Noah’s Ark Lab（华为诺亚实验室）
Dynamics Harmonic Analysis of Robotic Systems: Application in Data-Driven Koopman Modelling
Florida Institute for Human and Machine Cognition（佛罗里达人机认知研究所）；University College London（伦敦大学学院）；Istituto Italiano di Tecnologia（意大利技术研究院）；Heriot-Watt University（赫瑞瓦特大学）
DK-RRT: Deep Koopman RRT for Collision-Aware Motion Planning of Space Manipulators in Dynamic Debris Environments
University of California, Irvine（加州大学尔湾分校）；Illinois Institute of Technology（伊利诺伊理工学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Davis（加州大学戴维斯分校）；University of California, Berkeley（加州大学伯克利分校）
Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control
Istituto Italiano di Tecnologia（意大利技术研究院）；Institut de Robòtica i Informàtica Industrial（机器人与信息技术研究所）；MaLGa Center, DIBRIS, Università degli Studi di Genova（MaLGa中心，DIBRIS，热那亚大学）
Nonlinear dynamical systems can be made easier to control by lifting them into the space of observable functions, where their evolution is described by the linear Koopman operator. This paper describes how the Koopman operator can be used to generate approximate linear, bilinear, and nonlinear model realizations from data, and argues in favor of bilinear realizations for characterizing systems with unknown dynamics. Necessary and sufficient conditions for a dynamical system to have a valid linear or bilinear realization over a given set of observable functions are presented and used to show th…
Northwestern University（西北大学）；Michigan State University（密歇根州立大学）
This paper presents a generalizable methodology for data-driven identification of nonlinear dynamics that bounds the model error in terms of the prediction horizon and the magnitude of the derivatives of the system states. Using higher-order derivatives of general nonlinear dynamics that need not be known, we construct a Koopman operator-based linear representation and utilize Taylor series accuracy analysis to derive an error bound. The resulting error formula is used to choose the order of derivatives in the basis functions and obtain a data-driven Koopman model using a closed-form expressio…
Improving Robustness to Out-of-Distribution States in Imitation Learning via Deep Koopman-Boosted Diffusion Policy
Chair for Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich (TUM)（计算机辅助医疗程序与增强现实中心（CAMP），慕尼黑技术大学（TUM））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Medical Intelligence and Robotic Cognition Lab (MIRoC), The University of Hong Kong (HKU)（医疗智能与机器人认知实验室（MIRoC），香港大学（HKU））
Nonlinear dynamical effects are crucial to the operation of many agile robotic systems. Koopman-based model learning methods can capture these nonlinear dynamical system effects in higher dimensional lifted bilinear models that are amenable to optimal control. However, standard methods that lift the system state using a fixed function dictionary before model learning result in high dimensional models that are intractable for real time control. This paper presents a novel method that jointly learns a function dictionary and lifted bilinear model purely from data by incorporating the Koopman mod…
Zhejiang University（浙江大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Performing accurate localization while maintaining the low-level communication bandwidth is an essential challenge of multi-robot simultaneous localization and mapping (MR-SLAM). In this paper, we tackle this problem by generating a compact yet discriminative feature descriptor with minimum inference time. We propose descriptor distillation that formulates the descriptor generation into a learning problem under the teacher-student framework. To achieve real-time descriptor generation, we design a compact student network and learn it by transferring the knowledge from a pre-trained large teache…
Hubei University（湖北大学）；Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；Manchester Metropolitan Joint Institute, Hubei University（湖北大学曼城联合学院）
Data-Driven Predictive Control of Nonholonomic Robots Based on a Bilinear Koopman Realization: Data Does Not Replace Geometry
Institute of Engineering and Computational Mechanics (ITM), University of Stuttgart（工程与计算力学研究所（ITM），斯图加特大学）；Optimization-based Control Group, Institute of Mathematics, Technische Universität Ilmenau（优化控制组，数学研究所，伊尔梅瑙技术大学）；Department of Mechanical Engineering, LUT University（机械工程系，卢托大学）
