Active inference, a theoretical construct inspired by brain processing, is a promising alternative to control artificial agents. However, current methods do not yet scale to high-dimensional inputs in continuous control. Here we present a novel active inference torque controller for industrial arms that maintains the adaptive characteristics of previous proprioceptive approaches but also enables large-scale multimodal integration (e.g., raw images). We extended our previous mathematical formulation by including multimodal state representation learning using a linearly coupled multimodal variational autoencoder. We evaluated our model on a simulated 7DOF Franka Emika Panda robot arm and compared its behavior with a previous active inference baseline and the Panda built-in optimized controller. Results showed improved tracking and control in goal-directed reaching due to the increased representation power, high robustness to noise and adaptability in changes on the environmental conditions and robot parameters without the need to relearn the generative models nor parameters retuning.
University of Technology Sydney（悉尼科技大学）；South China University of Technology（华南理工大学）
Making sense of multiple modalities can yield a more comprehensive description of real-world phenomena. However, learning the co-representation of diverse modalities is still a long-standing endeavor in emerging machine learning applications and research. Previous generative approaches for multimodal input approximate a joint-modality posterior by uni-modality posteriors as product-of-experts (PoE) or mixture-of-experts (MoE). We argue that these approximations lead to a defective bound for the optimization process and loss of semantic connection among modalities. This paper presents a novel v…
Department of Computer Science, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；Committee on Computational and Applied Mathematics at the University of Chicago（芝加哥大学计算与应用数学系委员会）；Amazon Web Services（亚马逊云服务）
This paper reviews the novel concept of controllable variational autoencoder (ControlVAE), discusses its parameter tuning to meet application needs, derives its key analytic properties, and offers useful extensions and applications. ControlVAE is a new variational autoencoder (VAE) framework that combines the automatic control theory with the basic VAE to stabilize the KL-divergence of VAE models to a specified value. It leverages a non-linear PI controller, a variant of the proportional-integral-derivative (PID) control, to dynamically tune the weight of the KL-divergence term in the evidence…
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our po…
TU Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI））；Centre for Cognitive Science, TU Darmstadt（达姆施塔特工业大学认知科学中心）
Modeling interaction dynamics to generate robot trajectories that enable a robot to adapt and react to a human's actions and intentions is critical for efficient and effective collaborative Human-Robot Interactions (HRI). Learning from Demonstration (LfD) methods from Human-Human Interactions (HHI) have shown promising results, especially when coupled with representation learning techniques. However, such methods for learning HRI either do not scale well to high dimensional data or cannot accurately adapt to changing via-poses of the interacting partner. We propose Multimodal Interactive Laten…
The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem. We introduce a long short-term memory based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution. We also introduce an LSTM-VAE-based detector using a reconstruction-based anomaly score and a state-based threshold. For evaluations with 1,555 robot-assisted feeding exe…
Diffusion-Guided Semantic Consistency for Multimodal Heterogeneity
The University of British Columbia（不列颠哥伦比亚大学）；Fudan University（复旦大学）；Duke Kunshan University（杜克昆山大学）；East China Normal University（华东师范大学）；University of California, Davis（加州大学戴维斯分校）；China University of Mining and Technology（中国矿业大学）；SMBU；Shenzhen University（深圳大学）
Delft University of Technology（代尔夫特理工大学）；Radboud University（拉德堡德大学）
Adaptation to external and internal changes is major for robotic systems in uncertain environments. Here we present a novel multisensory active inference torque controller for industrial arms that shows how prediction can be used to resolve adaptation. Our controller, inspired by the predictive brain hypothesis, improves the capabilities of current active inference approaches by incorporating learning and multimodal integration of low and high-dimensional sensor inputs (e.g., raw images) while simplifying the architecture. We performed a systematic evaluation of our model on a 7DoF Franka Emik…
M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking
School of Automation, Southeast University（东南大学自动化学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动性学院）；Department of Robotics, Mohamed Bin Zayed University of Artificial Intelligence（马尔代夫比兹亚德大学人工智能学院机器人系）
TOP: Time Optimization Policy for Stable and Accurate Standing Manipulation with Humanoid Robots
The State Key Laboratory of Industrial Control and Technology of Zhejiang University（浙江大学工业控制技术国家重点实验室）；Department of Engineering Mechanics of Zhejiang University（浙江大学工程力学系）；Zhejiang Humanoid Robot Innovation Center（浙江人形机器人创新中心）
Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model
National Institute of Informatics（信息学研究所）；The Graduate University for Advanced Studies (SOKENDAI)（高级研究大学院大学（SOKENDAI））；Nara Institute of Science and Technology（奈良科学技术研究所）
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is non-trivial to manually design a robot controller that combines these modalities which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. In this work, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample eff…
