arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.03556cs.CVcs.RO
收藏

Afford-X:面向任务型操作的通用且轻量化的可供性推理模型

Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulation

Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Institute for Al Industry Research, Tsinghua University(清华大学人工智能产业研究院);Department of Computer Science, Tsinghua University(清华大学计算机科学系)

Xiaomeng Zhu, Yuyang Li, Leiyao Cui, Pengfei Li, Huan-ang Gao, Yixin Zhu, Hao Zhao

AI总结:

研究人员推出LVIS-Aff数据集,开发了Afford-X可供性推理模型,其性能优于现有非LLM方法和此前会议论文结果,参数紧凑、推理速度快,可部署于本地设备助力机器人任务导向型操作。

摘要:

物体可供性推理是指基于物理属性推断物体功能的能力,这是人类与人工智能(AI)进行任务导向型规划和活动的基础,该能力要求以任务导向的方式规划和执行日常活动,依赖于物体物理属性和功能的常识知识,超出了简单的物体识别范畴。当前从感知层面进行可供性推理的计算模型缺乏通用性,限制了其在新场景中的适用性;同时,具备新兴推理能力的综合性大语言模型(LLMs)难以部署在本地设备上用于任务导向型操作。在此,我们推出LVIS-Aff,这是一个包含1496个任务和11.9万张图像的大规模数据集,旨在提升从感知层面进行可供性推理的通用性。利用该数据集,我们开发了Afford-X,这是一种端到端可训练的可供性推理模型,整合了动词注意力(Verb Attention)和双融合(Bi-Fusion)模块以提升多模态理解能力。该模型相比已报道的非LLM方法的最佳结果实现了最高12.1%的性能提升,相比我们之前的会议论文结果也实现了1.2%的提升;同时它保持了紧凑的1.87亿参数规模,推理速度比GPT-4V API快近50倍。我们的工作证明了高效、通用的可供性推理模型的潜力,这类模型可部署在本地设备上用于任务导向型操作,我们展示了Afford-X在各类任务和环境中为机器人实现任务导向型操作的有效性,凸显了其高效性以及对推动机器人和AI系统在实际应用中发展的广泛意义。

英文摘要:

Object affordance reasoning, the ability to infer object functionalities based on physical properties, is fundamental for task-oriented planning and activities in both humans and Artificial Intelligence (AI). This capability, required for planning and executing daily activities in a task-oriented manner, relies on commonsense knowledge of object physics and functionalities, extending beyond simple object recognition. Current computational models for affordance reasoning from perception lack generalizability, limiting their applicability in novel scenarios. Meanwhile, comprehensive Large Language Models (LLMs) with emerging reasoning capabilities are challenging to deploy on local devices for task-oriented manipulations. Here, we introduce LVIS-Aff, a large-scale dataset comprising 1,496 tasks and 119k images, designed to enhance the generalizability of affordance reasoning from perception. Utilizing this dataset, we develop Afford-X, an end-to-end trainable affordance reasoning model that incorporates Verb Attention and Bi-Fusion modules to improve multi-modal understanding. This model achieves up to a 12.1% performance improvement over the best-reported results from non-LLM methods, while also demonstrating a 1.2% enhancement compared to our previous conference paper. Additionally, it maintains a compact 187M parameter size and infers nearly 50 times faster than the GPT-4V API. Our work demonstrates the potential for efficient, generalizable affordance reasoning models that can be deployed on local devices for task-oriented manipulations. We showcase Afford-X's effectiveness in enabling task-oriented manipulations for robots across various tasks and environments, underscoring its efficiency and broad implications for advancing robotics and AI systems in real-world applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑