Behavior prompting is a paradigm in which a sensorimotor robot demonstration, called a behavior prompt, serves as an in-context prompt for performing new tasks at test time. While prior work has shown that this capability is possible, the conditions that enable it remain poorly understood. We present an empirical study of when, how, and why behavior prompting works. To support this study, we introduce DrawAnything and LIBERO-Gen, benchmarks with up to 2000 procedurally generated tasks that evaluate test-time adaptation to unseen drawing and tabletop manipulation tasks. We also present Behavior Prompting Policy (BPP), an in-context visuomotor architecture, and iPhUMI, a handheld interface to demonstrate behavior prompts at test time. Our main finding is that task diversity, rather than demonstrations per task, is a key driver of prompting capability. Given sufficient diversity, a behavior prompt improves adaptation to unseen tasks, reducing drawing error by 80.7% over goal-image conditioning and improving success on chained manipulation tasks by up to 20.8% over language conditioning. Given insufficient diversity in a real-world laundry experiment, behavior prompting has weaker task conditioning than a language baseline. An attention analysis shows how the prompt is used: the policy follows it step by step as a source of dense sub-goals. Prompt ablations show that dense sensorimotor detail matters: removing actions or downsampling the prompt hurts fine-grained action adaptation. We have open-sourced all components to enable reproducible research on behavior prompting without needing industrial-scale data collection or compute.
AERMANI-VLM: Structured Prompting and Reasoning for Aerial Manipulation with Vision Language Models
Robotics Research Center, IIIT Hyderabad（IIIT海得拉巴机器人研究中心）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Newcastle University（纽卡斯尔大学）
Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Institute of Artificial Intelligence, China Telecom Corp Ltd（中国电信股份有限公司人工智能研究院）
MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulation
Nanyang Technological University（南洋理工大学）；VinUniversity（文大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；South China University of Technology（华南理工大学）
Robot manipulation requires a complex set of skills that need to be carefully combined and coordinated to solve a task. Yet, most ReinforcementLearning (RL) approaches in robotics study tasks which actually consist only of a single manipulation skill, such as grasping an object or inserting a pre-grasped object. As a result the skill ('how' to solve the task) but not the actual goal of a complete manipulation ('what' to solve) is specified. In contrast, we study a complex manipulation goal that requires an agent to learn and combine diverse manipulation skills. We propose a challenging, highly…
Robotic control tasks are often first run in simulation for the purposes of verification, debugging and data augmentation. Many methods exist to specify what task a robot must complete, but few exist to specify what range of environments a user expects such tasks to be achieved in. ProbRobScene is a probabilistic specification language for describing robotic manipulation environments. Using the language, a user need only specify the relational constraints that must hold between objects in a scene. ProbRobScene will then automatically generate scenes which conform to this specification. By comb…
Beyond Viewpoint Generalization: What Multi-View Demonstrations Offer and How to Synthesize Them for Robot Manipulation?
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Shenzhen University（深圳大学）；Beijing Jiaotong University（北京交通大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Choose What to Manipulate: Revealing Data Scaling Laws in Bounding-Box Guided Policies for Semantic Manipulation
Center for Intelligent Control and Telescience, Tsinghua Shenzhen International Graduate School（智能控制与遥科学中心，清华大学深圳国际研究生院）；Beijing Xiaomi Robot Technology Co., Ltd（北京小米机器人科技有限公司）
Extending the abilities of service robots is important for expanding what they can achieve in everyday manipulation tasks. On the other hand, it is also essential to ensure them to determine what they can not achieve in certain cases due to either anomalies or permanent failures during task execution. Robots need to identify these situations, and reveal the reasons behind these cases to overcome and recover from them. In this paper, we propose and analyze a Long Short-Term Memories-based (LSTM-based) awareness approach to reveal the reasons behind an anomaly case that occurs during a manipulat…
3D Affordance Keypoint Detection for Robotic Manipulation
Advanced Robotics Centre, National University of Singapore（新加坡国立大学先进机器人中心）；Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）
Contact Trajectory Prompting: In-Context Transfer of Contact-Rich Behaviors from a Single Demonstration
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；TARS Robotics（TARS机器人公司）；National University of Singapore（新加坡国立大学）
Stanford University（斯坦福大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most crit…
How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding nece…
Contact Modes Are Strata: What Geometric Structure Buys in Discrete-Continuous Planning
University of Toronto Institute for Aerospace Studies (UTIAS)（多伦多大学航空航天研究所）；Space and Terrestrial Autonomous Robotic Systems (STARS) Laboratory（空间与地面自主机器人系统实验室）
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
We are interested in pick-and-place style robot manipulation tasks in cluttered and confined 3D workspaces among movable objects that may be rearranged by the robot and may slide, tilt, lean or topple. A recently proposed algorithm, M4M, determines which objects need to be moved and where by solving a Multi-Agent Pathfinding MAPF abstraction of this problem. It then utilises a nonprehensile push planner to compute actions for how the robot might realise these rearrangements and a rigid body physics simulator to check whether the actions satisfy physics constraints encoded in the problem. Howev…
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Guangdong Provincial Key Laboratory of Computility Microelectronics, Faculty of Computility Microelectronics, Shenzhen University of Advanced Technology（深圳先进技术研究院计算机微科学与技术学院、广东省计算微电子重点实验室）；School of Aeronautics and Astronautics, Sichuan University（四川大学航空航天学院）；Suzhou Evans Intelligent Technology Co., Ltd.（苏州伊文斯智能科技有限公司）；Morphi Intelligence Technology Co., Ltd.（墨飞智能科技有限公司）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Xiaomi EV（小米汽车）
