Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulation
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Institute for Al Industry Research, Tsinghua University(清华大学人工智能产业研究院);Department of Computer Science, Tsinghua University(清华大学计算机科学系)
Object affordance reasoning, the ability to infer object functionalities based on physical properties, is fundamental for task-oriented planning and activities in both humans and Artificial Intelligence (AI). This capability, required for planning and executing daily activities in a task-oriented manner, relies on commonsense knowledge of object physics and functionalities, extending beyond simple object recognition. Current computational models for affordance reasoning from perception lack generalizability, limiting their applicability in novel scenarios. Meanwhile, comprehensive Large Language Models (LLMs) with emerging reasoning capabilities are challenging to deploy on local devices for task-oriented manipulations. Here, we introduce LVIS-Aff, a large-scale dataset comprising 1,496 tasks and 119k images, designed to enhance the generalizability of affordance reasoning from perception. Utilizing this dataset, we develop Afford-X, an end-to-end trainable affordance reasoning model that incorporates Verb Attention and Bi-Fusion modules to improve multi-modal understanding. This model achieves up to a 12.1% performance improvement over the best-reported results from non-LLM methods, while also demonstrating a 1.2% enhancement compared to our previous conference paper. Additionally, it maintains a compact 187M parameter size and infers nearly 50 times faster than the GPT-4V API. Our work demonstrates the potential for efficient, generalizable affordance reasoning models that can be deployed on local devices for task-oriented manipulations. We showcase Afford-X's effectiveness in enabling task-oriented manipulations for robots across various tasks and environments, underscoring its efficiency and broad implications for advancing robotics and AI systems in real-world applications.
RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室、计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation
Department of Computer Science, Rutgers University-New Brunswick(罗格斯大学新布朗斯维尔回声分校计算机科学系);The Hong Kong University of Science and Technology (GZ)(香港科学与技术大学(GZ));Shanghai AI Laboratory(上海人工智能实验室)
Language-Conditioned Affordance-Pose Detection in 3D Point Clouds
FPT Software AI Center(FPT软件人工智能中心);TU Wien(维也纳工业大学);Imperial College London(帝国理工学院);University of Arkansas(阿肯色大学);Ton Duc Thang University(孙德胜大学);University of Science, Vietnam(越南科学大学);University of Liverpool(利物浦大学)
Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions
CFCS, School of CS, PKU(计算机学院,北京大学);School of EECS, PKU(电子工程系,北京大学);University of Oxford(牛津大学);National Key Laboratory for Multimedia Information Processing, School of CS, PKU(多媒体信息处理国家重点实验室,计算机学院,北京大学)
Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance
Universität Hamburg(汉堡大学);Agile Robots AG(敏捷机器人公司);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
YOLOA: Real-Time Affordance Detection via LLM Adapter
School of Electronic Engineering, Xidian University(电子工程学院,西安电子科技大学);School of Software, Tsinghua University(软件学院,清华大学);School of Computer Science and Informatics, Cardiff University(计算机科学与信息学院,卡迪夫大学)
Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model
The Hong Kong University of Science and Technology (GZ)(香港科技大学(广州));National University of Singapore(新加坡国立大学);ShanghaiTech University(上海科技大学);East China Normal University(华东师范大学);Nanjing University of Information Science & Technology(南京信息工程大学);Zhejiang University(浙江大学);Institute of Automation, Chinese Academy of Science(中国科学院自动化研究所);Shanghai AI Laboratory(上海人工智能实验室)
Visual Affordance Prediction: Survey and Reproducibility
Istituto Italiano di Tecnologia(意大利理工研究院);Centre for Intelligent Sensing(智能感知中心);Queen Mary University of London(伦敦女王学院);Idiap Research Institute(Idiap研究 institute);EPFL(瑞士联邦理工学院)
RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping
The Chinese University of Hong Kong(香港中文大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Dexmal;Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);SKL-IOTSC, CIS, University of Macau(澳门科学馆-物联网与智能系统研究中心,澳门大学)
AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
The University of Hong Kong(香港大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学);National University of Singapore(新加坡国立大学);Knowin AI(考拉悠然科技)
Learning Affordance Grounding from Exocentric Images
JD Explore Academy(京东探索研究院);University of Science and Technology of China(中国科学技术大学);Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);The University of Sydney(悉尼大学)
University of Science and Technology of China(中国科学技术大学);The University of Sydney(悉尼大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
PartAfford: Part-level Affordance Discovery from 3D Objects
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Peking University(北京大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Tsinghua University(清华大学)
Grounding 3D Object Affordance from 2D Interactions in Images
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);University of Rochester(罗切斯特大学)