Beyond Viewpoint Generalization: What Multi-View Demonstrations Offer and How to Synthesize Them for Robot Manipulation?
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Shenzhen University（深圳大学）；Beijing Jiaotong University（北京交通大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Does multi-view demonstration truly improve robot manipulation, or merely enhance cross-view robustness? We present a systematic study quantifying the performance gains, scaling behavior, and underlying mechanisms of multi-view data for robot manipulation. Controlled experiments show that, under both fixed and randomized backgrounds, multi-view demonstrations consistently improve single-view policy success and generalization. Performance varies non-monotonically with view coverage, revealing effective regimes rather than a simple "more is better" trend. Notably, multi-view data breaks the scaling limitation of single-view datasets and continues to raise performance ceilings after saturation. Mechanistic analysis shows that multi-view learning promotes manipulation-relevant visual representations, better aligns the action head with the learned feature distribution, and reduces overfitting. Motivated by the importance of multi-view data and its scarcity in large-scale robotic datasets, as well as the difficulty of collecting additional viewpoints in real world settings, we propose RoboNVS, a geometry-aware self-supervised framework that synthesizes novel-view videos from monocular inputs. The generated data consistently improves downstream policies in both simulation and real-world environments.
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；Vector Institute（矢量研究所）
Learned visuomotor policies have shown considerable success as an alternative to traditional, hand-crafted frameworks for robotic manipulation. Surprisingly, an extension of these methods to the multiview domain is relatively unexplored. A successful multiview policy could be deployed on a mobile manipulation platform, allowing the robot to complete a task regardless of its view of the scene. In this work, we demonstrate that a multiview policy can be found through imitation learning by collecting data from a variety of viewpoints. We illustrate the general applicability of the method by learn…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）
We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations and use these skills to synthesize prolonged robot behaviors. Our method starts with constructing a hierarchical task structure from each demonstration through agglomerative clustering. From the task structures of multi-task demonstrations, we identify skills based on the recurring patterns and train goal-conditioned sensorimotor policies with hierarchical imitation learning. Finally, we train a meta control…
We present O2A, a novel method for learning to perform robotic manipulation tasks from a single (one-shot) third-person demonstration video. To our knowledge, it is the first time this has been done for a single demonstration. The key novelty lies in pre-training a feature extractor for creating a perceptual representation for actions that we call 'action vectors'. The action vectors are extracted using a 3D-CNN model pre-trained as an action classifier on a generic action dataset. The distance between the action vectors from the observed third-person demonstration and trial robot executions i…
EgoAVFlow: Robot Policy Learning with Active Vision from Human Egocentric Videos via 3D Flow
School of Interactive Computing, Georgia Institute of Technology（交互计算学院，佐治亚理工学院）；InnoCORE AI-Transformed Aerospace Research Center, KAIST（AI转型航空航天研究中心，韩国成均馆大学）
Expert demonstrations are a rich source of supervision for training visual robotic manipulation policies, but imitation learning methods often require either a large number of demonstrations or expensive online expert supervision to learn reactive closed-loop behaviors. In this work, we introduce SPARTN (Synthetic Perturbations for Augmenting Robot Trajectories via NeRF): a fully-offline data augmentation scheme for improving robot policies that use eye-in-hand cameras. Our approach leverages neural radiance fields (NeRFs) to synthetically inject corrective noise into visual demonstrations, us…
Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；the School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Rutgers University（罗格斯大学）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）
Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations
School of Computing and Data Science, The University of Hong Kong（计算科学与数据科学学院，香港大学）；LimX Dynamics Technology Co., Ltd（LimX动力技术有限公司）；Southern University of Science and Technology（南方科技大学）；Peking University（北京大学）；Zhejiang University（浙江大学）
EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning
Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）
Learning Generalizable 3D Manipulation With 10 Demonstrations
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所 机器人学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；College of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）
University of California, Berkeley（加州大学伯克利分校）；Idiap Research Institute（Idiap研究院）
Generalizing manipulation skills to new situations requires extracting invariant patterns from demonstrations. For example, the robot needs to understand the demonstrations at a higher level while being invariant to the appearance of the objects, geometric aspects of objects such as its position, size, orientation and viewpoint of the observer in the demonstrations. In this paper, we propose an algorithm that learns a joint probability density function of the demonstrations with invariant formulations of hidden semi-Markov models to extract invariant segments (also termed as sub-goals or optio…
AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Department of Mechanical Engineering, Massachusetts Institute of Technology（机械工程系，麻省理工学院）
Co-training with Ego-centric Video and Demonstration for Robot Navigation Task
Department of Informatics, Graduate School of Informatics, Kyoto University（信息学系，京都大学研究生院）；Spatial Robotics Research Center, Fujitsu Limited（空间机器人研究中心，富士通有限公司）
Learning from Demonstration (LfD) seeks to democratize robotics by enabling non-roboticist end-users to teach robots to perform a task by providing a human demonstration. However, modern LfD techniques, e.g. inverse reinforcement learning (IRL), assume users provide at least stochastically optimal demonstrations. This assumption fails to hold in most real-world scenarios. Recent attempts to learn from sub-optimal demonstration leverage pairwise rankings and following the Luce-Shepard rule. However, we show these approaches make incorrect assumptions and thus suffer from brittle, degraded perfo…
SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining
Institute of Artificial Intelligence, China Telecom（中国电信人工智能研究院）；Technical University of Munich（慕尼黑工业大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）
Grasp as You Dream: Imitating Functional Grasping from Generated Human Demonstrations
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology（KTH皇家理工学院机器人、感知与学习系）；Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology（南方科技大学深圳市机器人视觉与感知重点实验室）
WM-DAgger: Enabling Efficient Data Aggregation for Imitation Learning with World Models
Peking University（北京大学）；JD Logistics（京东物流）；Nankai University（南开大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Rutgers University（罗格斯大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；Institut Polytechnique de Paris（巴黎综合理工学院）
In this work we propose a novel end-to-end imitation learning approach which combines natural language, vision, and motion information to produce an abstract representation of a task, which in turn is used to synthesize specific motion controllers at run-time. This multimodal approach enables generalization to a wide variety of environmental conditions and allows an end-user to direct a robot policy through verbal communication. We empirically validate our approach with an extensive set of simulations and show that it achieves a high task success rate over a variety of conditions while remaini…
In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method on 100 hours of expert demonstrations on urb…
Grasping and manipulating objects is an important human skill. Since most objects are designed to be manipulated by human hands, anthropomorphic hands can enable richer human-robot interaction. Desirable grasps are not only stable, but also functional: they enable post-grasp actions with the object. However, functional grasp synthesis for high degree-of-freedom anthropomorphic hands from object shape alone is challenging because of the large optimization space. We present ContactGrasp, a framework for functional grasp synthesis from object shape and contact on the object surface. Contact can b…
