This paper presents a novel structured knowledge representation called the functional object-oriented network (FOON) to model the connectivity of the functional-related objects and their motions in manipulation tasks. The graphical model FOON is learned by observing object state change and human manipulations with the objects. Using a well-trained FOON, robots can decipher a task goal, seek the correct objects at the desired states on which to operate, and generate a sequence of proper manipulation motions. The paper describes FOON's structure and an approach to form a universal FOON with extracted knowledge from online instructional videos. A graph retrieval approach is presented to generate manipulation motion sequences from the FOON to achieve a desired goal, demonstrating the flexibility of FOON in creating a novel and adaptive means of solving a problem using knowledge gathered from multiple sources. The results are demonstrated in a simulated environment to illustrate the motion sequences generated from the FOON to carry out the desired tasks.
DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo
NLPR & MAIS, CASIA(中国科学院南京自动化研究所与模式识别与人工智能实验室);SJTU(上海交通大学);MBZUAI(马克斯·普朗克研究所);Beijing Institute of Basic Medical Sciences(北京基础医学研究所);PKU & Galbot(北京大学与Galbot);CUHK(香港大学)
A4A: Cross-Embodiment Transfer of Action-Oriented 4D Affordances from Human Demonstrations
Shanghai Jiao Tong University(上海交通大学);Rutgers University–New Brunswick(罗格斯大学新布朗斯维克分校);Nanyang Technological University(南洋理工大学);The Hong Kong University of Science and Technology (GZ)(香港科技大学(广州));Shanghai AI Laboratory(上海人工智能实验室)
TONAV: Task-Oriented Navigation and Action-Velocity Chunk Learning for Articulated Object Quadrupedal Mobile Manipulation
School of Artificial Intelligence and Robotics, Hunan University(湖南大学人工智能与机器人学院);National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(湖南大学机器人视觉感知与控制技术国家工程研究中心);College of Semiconductors (College of Integrated Circuits), Hunan University(湖南大学半导体学院(集成电路学院))
Manipulate as Human: Learning Task-oriented Manipulation Skills by Adversarial Motion Priors
ParisTech Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎高科精英技术研究院);Department of Automation, Shanghai Jiao Tong University(上海交通大学自动化学院);MoE Key Lab of Artificial Intelligence and AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能与AI研究院)
PartDexTOG: Generating Dexterous Task-Oriented Grasping via Language-driven Part Analysis
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)
Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulation
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Institute for Al Industry Research, Tsinghua University(清华大学人工智能产业研究院);Department of Computer Science, Tsinghua University(清华大学计算机科学系)
AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Beijing Academy of Artificial Intelligence(北京人工智能研究院);Peking University(北京大学)
Prioritized Planning for Target-Oriented Manipulation via Hierarchical Stacking Relationship Prediction
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);National Engineering Research Center for Visual Information and Application(视觉信息与应用国家工程研究中心);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究所);Xi’an Jiaotong University(西安交通大学)
GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);Tsinghua University(清华大学);Fudan University(复旦大学)
该研究提出 GIFT 框架,通过几何对齐、可供性预测和目标区域重建约束中间特征,在 LIBERO-Plus 和 RoboCasa 数据集上,其多个变体在机器人操作任务中显著优于基准模型,性能提升明显。
DexTOG: Learning Task-Oriented Dexterous Grasp with Language
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);AI Institute, Shanghai Jiao Tong University(上海交通大学AI研究院);Southeast University(东南大学)
ROG-Grasp: Root-Oriented Geometry for Robotic Grasping and Placement
Department of Electrical and Computer Engineering, Drexel University(德雷塞尔大学电气与计算机工程系);Virginia Seafood Agricultural Research and Extension Center, and Department of Biological Systems Engineering, Virginia Tech(弗吉尼亚理工学院生物系统工程系和弗吉尼亚海鲜农业研究与推广中心);Amazon Store Foundation AI (SFAI)(亚马逊商店基金会人工智能(SFAI))
The functional and temporal roles of gaze evolve across the phases and constraints of multi-stage robot-mediated manipulation
The BioRobotics Institute, Scuola Superiore Sant’Anna(生物机器人研究所,高等圣安娜学院);Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna(机器人与人工智能卓越部门,高等圣安娜学院)
Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);Nanjing University(南京大学);Central South University(中南大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Normal University(北京师范大学)