我们提出了一个辅助机器人操作框架，重点关注两个基本挑战：第一，高效地将大规模模型适配到下游场景可供性理解任务中，尤其是在日常生活场景中，收集涉及人类的多任务数据需要付出巨大努力；第二，通过将视觉可供性模型接地来有效学习机器人动作轨迹。我们通过采用参数高效的提示微调方法应对第一个挑战，该方法将可学习的文本提示前置到冻结的视觉模型中，以在多任务场景中预测操作可供性。然后，我们提出在监督流匹配方法中，以可供性为引导学习机器人动作轨迹。流匹配将机器人视觉运动策略表示为将随机路径点流向期望机器人动作轨迹的条件过程。最后，我们引入了一个包含日常生活活动（Activities of Daily Living）中10个任务的真实世界数据集来测试我们的框架。我们广泛的评估表明，所提出的用于学习操作可供性的提示微调方法在不同数据规模下取得了有竞争力的性能，甚至优于其他一些微调协议，同时满足参数效率。使用流匹配学习多任务机器人动作轨迹在几个机器人操作基准上比一些替代的行为克隆方法取得了一致有利的结果。这包括更稳定的训练和评估，以及明显更快的推理，同时保持与扩散策略相当的泛化性能，在大多数情况下流匹配表现略好。我们的框架将可供性学习与动作生成通过流匹配无缝统一用于机器人操作。
英文摘要：
We present a framework for assistive robot manipulation, which focuses on two fundamental challenges: first, efficiently adapting large-scale models to downstream scene affordance understanding tasks, especially in daily living scenarios where gathering multi-task data involving humans requires strenuous effort; second, effectively learning robot action trajectories by grounding the visual affordance model. We tackle the first challenge by employing a parameter-efficient prompt tuning method that prepends learnable text prompts to the frozen vision model to predict manipulation affordances in multi-task scenarios. Then we propose to learn robot action trajectories guided by affordances in a supervised flow matching method. Flow matching represents a robot visuomotor policy as a conditional process of flowing random waypoints to desired robot action trajectories. Finally, we introduce a real-world dataset with 10 tasks across Activities of Daily Living to test our framework. Our extensive evaluation highlights that the proposed prompt tuning method for learning manipulation affordance achieves competitive performance and even outperforms some other finetuning protocols across data scales, while satisfying parameter efficiency. Learning multi-task robot action trajectories with flow matching leads to consistently favorable results in several robot manipulation benchmarks than some alternative behavior cloning methods. This includes more stable training and evaluation, and noticeably faster inference, while maintaining comparable generalization performance to diffusion policy, where flow matching performs marginally better in most cases. Our framework seamlessly unifies affordance learning and action generation with flow matching for robot manipulation.
Learning to manipulate 3D objects in an interactive environment has been a challenging problem in Reinforcement Learning (RL). In particular, it is hard to train a policy that can generalize over objects with different semantic categories, diverse shape geometry and versatile functionality. Recently, the technique of visual affordance has shown great prospects in providing object-centric information priors with effective actionable semantics. As such, an effective policy can be trained to open a door by knowing how to exert force on the handle. However, to learn the affordance, it often requir…
We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closed-loop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable obj…
AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Knowin AI（考拉悠然科技）
University of Liverpool（利物浦大学）；King’s College London（伦敦国王学院）；Monash University（莫纳什大学）
Transparent objects are widely used in our daily lives and therefore robots need to be able to handle them. However, transparent objects suffer from light reflection and refraction, which makes it challenging to obtain the accurate depth maps required to perform handling tasks. In this paper, we propose a novel affordance-based framework for depth reconstruction and manipulation of transparent objects, named A4T. A hierarchical AffordanceNet is first used to detect the transparent objects and their associated affordances that encode the relative positions of an object's different parts. Then, …
AnchorDP3: 3D Affordance Guided Sparse Diffusion Policy for Robotic Manipulation
Jingdong Technology Information Technology Co., Ltd（京东科技信息技术有限公司）；Tsinghua University（清华大学）；Fudan University（复旦大学）；Southeast University（东南大学）；University of Science and Technology of China（中国科学技术大学）
DORA: Object Affordance-Guided Reinforcement Learning for Dexterous Robotic Manipulation
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, Universität Hamburg（技术多模态系统（TAMS），信息学院，汉堡大学）；Agile Robots AG（敏捷机器人公司）；Technical University of Munich（慕尼黑技术大学）
MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulation
Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Northwestern Polytechnical University（西北工业大学）；TeleAI, China Telecom Corp Ltd（中国电信 TeleAI 公司）
3D Affordance Keypoint Detection for Robotic Manipulation
Advanced Robotics Centre, National University of Singapore（新加坡国立大学先进机器人中心）；Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）
School of CS & AI, Arizona State University（亚利桑那州立大学计算机科学与人工智能学院）
Conventional works that learn grasping affordance from demonstrations need to explicitly predict grasping configurations, such as gripper approaching angles or grasping preshapes. Classic motion planners could then sample trajectories by using such predicted configurations. In this work, our goal is instead to fill the gap between affordance discovery and affordance-based policy learning by integrating the two objectives in an end-to-end imitation learning framework based on deep neural networks. From a psychological perspective, there is a close association between attention and affordance. T…
A*STAR Institute for Infocomm Research (IR)（A*STAR资讯通信研究院（IR））；A*STAR Institute of High Performance Computing (IHPC)（A*STAR高性能计算研究院（IHPC））
Robots are increasingly expected to manipulate objects in ever more unstructured environments where the object properties have high perceptual uncertainty from any single sensory modality. This directly impacts successful object manipulation. In this work, we propose a reinforcement learning-based motion planning framework for object manipulation which makes use of both on-the-fly multisensory feedback and a learned attention-guided deep affordance model as perceptual states. The affordance model is learned from multiple sensory modalities, including vision and touch (tactile and force/torque)…
Affordance information about a scene provides important clues as to what actions may be executed in pursuit of meeting a specified goal state. Thus, integrating affordance-based reasoning into symbolic action plannning pipelines would enhance the flexibility of robot manipulation. Unfortunately, the top performing affordance recognition methods use object category priors to boost the accuracy of affordance detection and segmentation. Object priors limit generalization to unknown object categories. This paper describes an affordance recognition pipeline based on a category-agnostic region propo…
RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
Xiaomi EV（小米电动车）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学 Gallagher人工智能学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）
University of Michigan（密歇根大学）；Robotics Institute（机器人研究所）
In order to enable robust operation in unstructured environments, robots should be able to generalize manipulation actions to novel object instances. For example, to pour and serve a drink, a robot should be able to recognize novel containers which afford the task. Most importantly, robots should be able to manipulate these novel containers to fulfill the task. To achieve this, we aim to provide robust and generalized perception of object affordances and their associated manipulation poses for reliable manipulation. In this work, we combine the notions of affordance and category-level pose, an…
IEETA - University of Aveiro（阿威罗大学电子与电信研究所）；University of Groningen（格罗宁根大学）
Service robots are expected to autonomously and efficiently work in human-centric environments. For this type of robots, object perception and manipulation are challenging tasks due to need for accurate and real-time response. This paper presents an interactive open-ended learning approach to recognize multiple objects and their grasp affordances concurrently. This is an important contribution in the field of service robots since no matter how extensive the training data used for batch learning, a robot might always be confronted with an unknown object when operating in human-centric environme…
Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning
Department of Computer Science, Technical University of Darmstadt（达姆施塔特工业大学计算机科学系）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, Technical University of Darmstadt（达姆施塔特工业大学认知科学中心）；Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt（黑森州人工智能中心）；Robotics Institute Germany (RIG)（德国机器人研究所）
UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models
Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）；Rutgers University（罗格斯大学）；Hefei University of Technology（合肥工业大学）；Shanghai AI Lab（上海人工智能实验室）
University of California, Los Angeles（加州大学洛杉矶分校）；Indiana University, Bloomington（印第安纳大学伯明顿分校）
In this paper, we present an approach for robot learning of social affordance from human activity videos. We consider the problem in the context of human-robot interaction: Our approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event (i.e., sub-goal). This enables the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions. We introduce the rep…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Fudan University（复旦大学）；Indiana University（印第安纳大学）
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI). Based on Gibbs sampling, our weakly supervised grammar learning can automatically construct a hierarchical representation of an interaction with long-term joint sub-tasks of both agents and short term atomic actions of individual agents. Based on a new RGB-D video dataset with rich instances of human interactions, our …
Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics. While recent progress in imitation learning has largely improved the sample efficiency compared to Reinforcement Learning, the learned policy can hardly generalize to manipulate novel objects, given limited expert demonstrations. In this paper, we propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category, which are generated from a human grasp affordance model. This generalizes the policy to novel object instances within the same category. To…
Perception-for-grasping is a challenging problem in robotics. Inexpensive range sensors such as the Microsoft Kinect provide sensing capabilities that have given new life to the effort of developing robust and accurate perception methods for robot grasping. This paper proposes a new approach to localizing enveloping grasp affordances in 3-D point clouds efficiently. The approach is based on modeling enveloping grasp affordances as a cylindrical shells that corresponds to the geometry of the robot hand. A fast and accurate fitting method for quadratic surfaces is the core of our approach. An ev…
Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等理工学院）；Champalimaud Centre for the Unknown（尚帕利莫未知中心）；Queen Mary University of London（伦敦玛丽女王大学）
One of the open challenges in designing robots that operate successfully in the unpredictable human environment is how to make them able to predict what actions they can perform on objects, and what their effects will be, i.e., the ability to perceive object affordances. Since modeling all the possible world interactions is unfeasible, learning from experience is required, posing the challenge of collecting a large amount of experiences (i.e., training data). Typically, a manipulative robot operates on external objects by using its own hands (or similar end-effectors), but in some cases the us…
Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等理工学院）；Queen Mary University of London（伦敦玛丽女王大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
A growing field in robotics and Artificial Intelligence (AI) research is human-robot collaboration, whose target is to enable effective teamwork between humans and robots. However, in many situations human teams are still superior to human-robot teams, primarily because human teams can easily agree on a common goal with language, and the individual members observe each other effectively, leveraging their shared motor repertoire and sensorimotor resources. This paper shows that for cognitive robots it is possible, and indeed fruitful, to combine knowledge acquired from interacting with elements…
Open containers, i.e., containers without covers, are an important and ubiquitous class of objects in human life. In this letter, we propose a novel method for robots to "imagine" the open containability affordance of a previously unseen object via physical simulations. We implement our imagination method on a UR5 manipulator. The robot autonomously scans the object with an RGB-D camera. The scanned 3D model is used for open containability imagination which quantifies the open containability affordance by physically simulating dropping particles onto the object and counting how many particles …
PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation
University of Pennsylvania（宾夕法尼亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Nanyang Technological University（南洋理工大学）；University of Oxford（牛津大学）；Massachusetts Institute of Technology（麻省理工学院）
RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室、计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
