Can we learn robot manipulation for everyday tasks, only by watching videos of humans doing arbitrary tasks in different unstructured settings? Unlike widely adopted strategies of learning task-specific behaviors or direct imitation of a human video, we develop a a framework for extracting agent-agnostic action representations from human videos, and then map it to the agent's embodiment during deployment. Our framework is based on predicting plausible human hand trajectories given an initial image of a scene. After training this prediction model on a diverse set of human videos from the internet, we deploy the trained model zero-shot for physical robot manipulation tasks, after appropriate transformations to the robot's embodiment. This simple strategy lets us solve coarse manipulation tasks like opening and closing drawers, pushing, and tool use, without access to any in-domain robot manipulation trajectories. Our real-world deployment results establish a strong baseline for action prediction information that can be acquired from diverse arbitrary videos of human activities, and be useful for zero-shot robotic manipulation in unseen scenes.
Khoury College of Computer Sciences, Northeastern University（东北大学库里计算机科学学院）；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所）；Google Research, Brain Team（谷歌研究院大脑团队）
World models for environments with many objects face a combinatorial explosion of states: as the number of objects increases, the number of possible arrangements grows exponentially. In this paper, we learn to generalize over robotic pick-and-place tasks using object-factored world models, which combat the combinatorial explosion by ensuring that predictions are equivariant to permutations of objects. Previous object-factored models were limited either by their inability to model actions, or by their inability to plan for complex manipulation tasks. We build on recent contrastive methods for t…
MoTo: A Zero-shot Plug-in Interaction-aware Navigation for General Mobile Manipulation
Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；Beijing Key Laboratory of Embodied Intelligence Systems（北京市具身智能系统重点实验室）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Nanyang Technological University（南洋理工大学）
The vast majority of visual animals actively control their eyes, heads, and/or bodies to direct their gaze toward different parts of their environment. In contrast, recent applications of reinforcement learning in robotic manipulation employ cameras as passive sensors. These are carefully placed to view a scene from a fixed pose. Active perception allows animals to gather the most relevant information about the world and focus their computational resources where needed. It also enables them to view objects from different distances and viewpoints, providing a rich visual experience from which t…
Goal-VLA: Image-Generative VLMs as Object-Centric World Models Empowering Zero-shot Robot Manipulation
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；The HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong（香港大学数据科学研究院）；Yuanpei College, Peking University（北京大学元培学院）；Department of Automation, Tsinghua University（清华大学自动化系）
UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；Department of Automation, Tsinghua University, China（清华大学自动化系）
We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of …
This paper introduces the i-HY Hand, an underactuated hand driven by 5 actuators that is capable of performing a wide range of grasping and in-hand manipulation tasks. This hand was designed to address the need for a durable, inexpensive, moderately dexterous hand suitable for use on mobile robots. The primary focus of this paper will be on the novel minimalistic design of i-HY, which was developed by choosing a set of target tasks around which the design of the hand was optimized. Particular emphasis is placed on the development of underactuated fingers that are capable of both firm power gra…
SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation
CFCS, School of Computer Science, Peking University（北京大学计算机学院CFCS）；Institute for AI, Peking University（北京大学人工智能研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；PKU-WUHAN Institute for Artificial Intelligence, China（北京大学武汉人工智能研究院）
From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation
Shanghai Jiao Tong University（上海交通大学）；Northeastern University（东北大学）；Xiamen University Malaysia（厦门大学马来西亚分校）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Xspark AI
Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation
National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（国家关键人工智能实验室，北京通用人工智能研究院（BIGAI））；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京行为与心理健康重点实验室，北京大学）；Yuanpei College, Peking University（北京大学元培学院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，北京大学-武汉人工智能研究院）
[...] We argue that the traditional grasp modeling theory assumes a complexity that most robotic hands do not possess and is therefore of limited applicability to the robotic hands commonly used today. We discuss these limitations of the existing grasp modeling literature and set out to develop our own tools for the analysis of passive effects in robotic grasping. We show that problems of this kind are difficult to solve due to the non-convexity of the Maximum Dissipation Principle (MDP), which is part of the Coulomb friction law. We show that for planar grasps the MDP can be decomposed into a…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（脸书人工智能研究院）
Task specification is at the core of programming autonomous robots. A low-effort modality for task specification is critical for engagement of non-expert end-users and ultimate adoption of personalized robot agents. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. The former is hard to interpret for non-experts and necessitates detailed state estimation and scene understanding. The latter requires the generation of desired goal image, which often requires a human to complete the task, defeating the pu…
Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation
Embodied AI and Robotics (AIR) Lab, New York University, New York, USA and NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE（纽约大学Embodied AI和机器人实验室及纽约大学阿布扎赫尔人工智能与机器人中心）；Harvard AI and Robotics Lab, Harvard University, Boston, USA（哈佛大学人工智能与机器人实验室）；Department of Computer Science, University College London, London, UK（伦敦大学学院计算机科学系）；Department of Computer Science, University of Liverpool, UK（利物浦大学计算机科学系）；NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE（纽约大学阿布扎赫尔人工智能与机器人中心）
SPARK Hand: Scooping-Pinching Adaptive Robotic Hand with Kempe Mechanism for Vertical Passive Grasp in Environmental Constraints
School of Future Technology, Harbin Institute of Technology（哈尔滨工业大学未来技术学院）；Laboratory of Robotics, X-Institute（X研究所机器人实验室）；Shude College, Southern University of Science and Technology（南方科技大学舍德学院）；Department of Mechanical Engineering, Tsinghua University（清华大学机械工程系）
A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University（机器人与自动信息系统研究所，人工智能学院，南开大学）；Tianjin Key Laboratory of Intelligent Robotics, Nankai University（智能机器人重点实验室，南开大学）；College of Artificial Intelligence, Nankai University（人工智能学院，南开大学）；Department of Computing, The Hong Kong Polytechnic University（computing 部门，香港理工大学）
High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting
Wuhan University（武汉大学）；DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Hupan Lab（虎扑实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）；Zhejiang University（浙江大学）
The sense of touch is fundamental in several manipulation tasks, but rarely used in robot manipulation. In this work we tackle the problem of learning rich touch features from cross-modal self-supervision. We evaluate them identifying objects and their properties in a few-shot classification setting. Two new datasets are introduced using a simulated anthropomorphic robotic hand equipped with tactile sensors on both synthetic and daily life objects. Several self-supervised learning methods are benchmarked on these datasets, by evaluating few-shot classification on unseen objects and poses. Our …
UniLM-Nav: A Unified Framework for Zero-Shot Last-Mile Navigation
Tsinghua University（清华大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，字节跳动公司人工智能研究院）；Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）
The past decade has witnessed the tremendous successes of machine learning techniques in the supervised learning paradigm, where there is a clear demarcation between training and testing. In the supervised learning paradigm, learning is inherently passive, seeking to distill human-provided supervision in large-scale datasets into high capacity models. Following these successes, machine learning researchers have looked beyond this paradigm and became interested in tasks that are more dynamic. To them, robotics serve as an excellent test-bed, for the challenges of robotics break many of the assu…
Morphologically Equivariant Flow Matching for Bimanual Mobile Manipulation
PEARL Lab, Dept. of Computer Science, TU Darmstadt, Germany（德累斯顿技术大学计算机科学系PEARL实验室）；Istituto Italiano di Tecnologia（意大利技术研究院）；Robotics Institute Germany（德国机器人研究所）
