Existing robotic manipulation methods primarily rely on visual and proprioceptive observations, which may struggle to infer contact-related interaction states in partially observable real-world environments. Acoustic cues, by contrast, naturally encode rich interaction dynamics during contact, yet remain underexploited in current multimodal fusion literature. Most multimodal fusion approaches implicitly assume homogeneous roles across modalities, and thus design flat and symmetric fusion structures. However, this assumption is ill-suited for acoustic signals, which are inherently sparse and contact-driven. To achieve precise robotic manipulation through acoustic-informed perception, we propose a hierarchical representation fusion framework that progressively integrates audio, vision, and proprioception. Our approach first conditions visual and proprioceptive representations on acoustic cues, and then explicitly models higher-order cross-modal interactions to capture complementary dependencies among modalities. The fused representation is leveraged by a diffusion-based policy to directly generate continuous robot actions from multimodal observations. The combination of end-to-end learning and hierarchical fusion structure enables the policy to exploit task-relevant acoustic information while mitigating interference from less informative modalities. The proposed method has been evaluated on real-world robotic manipulation tasks, including liquid pouring and cabinet opening. Extensive experiment results demonstrate that our approach consistently outperforms state-of-the-art multimodal fusion frameworks, particularly in scenarios where acoustic cues provide task-relevant information not readily available from visual observations alone. Furthermore, a mutual information analysis is conducted to interpret the effect of audio cues in robotic manipulation via multimodal fusion.
Whole-Body Proprioceptive Morphing: A Modular Soft Gripper for Robust Cross-Scale Grasping
Mechanical Engineering Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机械工程系）；Robotics Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机器人学系）
ProWorld: Multi-View World Model with Proprioceptive Feedback for Closed-Loop Policy Evaluation
Sun Yat-sen University（中山大学）；University of Macau（澳门大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Meituan（美团）；South China University of Technology（华南理工大学）
Object Classification Utilizing Neuromorphic Proprioceptive Signals in Active Exploration: Validated on a Soft Anthropomorphic Hand
Institute for Cognitive Systems, Technical University of Munich（认知系统研究所，慕尼黑技术大学）；Department of Biomedical Engineering, Johns Hopkins University（生物医学工程系，约翰霍普金斯大学）
Referring expressions are commonly used when referring to a specific target in people's daily dialogue. In this paper, we develop a novel task of audio-visual grounding referring expression for robotic manipulation. The robot leverages both the audio and visual information to understand the referring expression in the given manipulation instruction and the corresponding manipulations are implemented. To solve the proposed task, an audio-visual framework is proposed for visual localization and sound recognition. We have also established a dataset which contains visual data, auditory data and ma…
Finger-gaiting manipulation is an important skill to achieve large-angle in-hand re-orientation of objects. However, achieving these gaits with arbitrary orientations of the hand is challenging due to the unstable nature of the task. In this work, we use model-free reinforcement learning (RL) to learn finger-gaiting only via precision grasps and demonstrate finger-gaiting for rotation about an axis purely using on-board proprioceptive and tactile feedback. To tackle the inherent instability of precision grasping, we propose the use of initial state distributions that enable effective explorati…
Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets
University of California, San Diego（加利福尼亚大学圣迭戈分校）；Tongji University（同济大学）；Shanghai Jiao Tong University（上海交通大学）；University of Maryland, College Park（马里兰大学帕克分校）；Tsinghua University（清华大学）
Robot safety has been a prominent research topic in recent years since robots are more involved in daily tasks. It is crucial to devise the required safety mechanisms to enable service robots to be aware of and react to anomalies (i.e., unexpected deviations from intended outcomes) that arise during the execution of these tasks. Detection and identification of these anomalies is an essential step towards fulfilling these requirements. Although several architectures are proposed for anomaly detection, identification is not yet thoroughly investigated. This task is challenging since indicators m…
RoboMatch: A Unified Mobile-Manipulation Teleoperation Platform with Auto-Matching Network Architecture for Long-Horizon Tasks
Jiangsu Key Laboratory of Advanced Food Manufacturing Equipment and Technology, School of Mechanical Engineering, Jiangnan University（江苏省食品先进制造装备与技术重点实验室，江南大学机械工程学院）
The University of Melbourne（墨尔本大学）；St. Vincent’s Hospital（圣文森特医院）
Tactile-based blind grasping addresses realistic robotic grasping in which the hand only has access to proprioceptive and tactile sensors. The robotic hand has no prior knowledge of the object/grasp properties, such as object weight, inertia, and shape. There exists no manipulation controller that rigorously guarantees object manipulation in such a setting. Here, a robust control law is proposed for object manipulation in tactile-based blind grasping. The analysis ensures semi-global asymptotic and exponential stability in the presence of model uncertainties and external disturbances that are …
When would Vision-Proprioception Policies Fail in Robotic Manipulation?
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学光明学院人工智能学院）；Beijing Key Laboratory of Research on Large Models（北京大模型与智能治理研究重点实验室）；School of Artificial Intelligence, Beihang University（北航人工智能学院）
Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasp…
Online Estimation and Manipulation of Articulated Objects
RIPL-Lab, University of Waterloo, Canada（滑铁卢大学）；Robot Learning Lab, University of Freiburg, Germany（弗赖堡大学）；School of Informatics, University of Edinburgh, UK（爱丁堡大学）
Tsinghua University（清华大学）；University of Edinburgh（爱丁堡大学）
In robotic grasping, objects are often occluded in ungraspable configurations such that no pregrasp pose can be found, eg large flat boxes on the table that can only be grasped from the side. Inspired by humans' bimanual manipulation, eg one hand to lift up things and the other to grasp, we address this type of problems by introducing pregrasp manipulation - push and lift actions. We propose a model-free Deep Reinforcement Learning framework to train control policies that utilize visual information and proprioceptive states of the robot to autonomously discover robust pregrasp manipulation. Th…
FWBC-VLA: Force-Aware Whole-Body Compensation for Contact-Rich Loco-Manipulation
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Zhongguancun Academy（中关村学院）；Deep Robotics（深度机器人公司）；Zhejiang University of Science and Technology（浙江科技大学）
Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and Shanghai Key Laboratory of Navigation and Location Based Services（自动化与智能感知学院，上海交通大学，导航与基于位置的服务重点实验室）；Department of Engineering, Cambridge University（工程系，剑桥大学）
ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences (CAS)（中国科学院大学）；Beijing Academy of Artificial Intelligence (BAAI)（北京智源人工智能研究院）
M4Diffuser: Multi-View Diffusion Policy with Manipulability-Aware Control for Robust Mobile Manipulation
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg（汉堡大学信息学院技术多模态系统部门）；Technical University of Munich（慕尼黑技术大学）；Agile Robots SE（敏捷机器人有限公司）
Blind Dexterity: Whole-Body Humanoid Manipulation via Pure Proprioception
TU Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Tongji University（同济大学）；Shanghai Research Institute for Autonomous Intelligent Systems（上海自主智能系统研究院）；Robotics Institute Germany（德国机器人研究所）
