arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.29419cs.ROcs.AIcs.CV
收藏

RAAP:基于跨图像动作对齐的检索增强的可达性预测

RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University(东南大学计算机科学与工程学院、新一代人工智能技术与交叉应用重点实验室);Southeast University-Monash University Joint Graduate School, Southeast University(东南大学-蒙纳士大学联合研究生院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)

Qiyuan Zhuang, He-Yang Xu, Yijun Wang, Xin-Yang Zhao, Yang-Yang Li, Xiu-Shen Wei

AI总结:

RAAP通过结合检索与对齐学习,统一了可达性检索与动作方向预测,利用密集对应转移接触点并预测动作方向,实现了跨未见物体和类别的稳健泛化。

摘要:

理解物体可达性对于使机器人在多样且无结构的环境中执行有目的的精细交互至关重要。然而,现有方法要么依赖检索,因稀疏性和覆盖间隙而脆弱,要么依赖大规模模型,当应用于未见类别时常误定位接触点并误预测接触后动作,从而阻碍了稳健泛化。我们介绍了检索增强的可达性预测(RAAP),一种将可达性检索与基于对齐的学习统一的框架。通过解耦静态接触定位与动态动作方向,RAAP通过密集对应转移接触点,并通过检索增强的对齐模型预测动作方向,该模型通过双加权注意力整合多个参考。在DROID和HOI4D的紧凑子集中训练,仅需每个任务数十个样本,RAAP在未见物体和类别上实现了一致性能,并在仿真和现实世界中实现了零样本机器人操作。项目网站:https://github.com/SEU-VIPGroup/RAAP。

英文摘要:

Understanding object affordances is essential for enabling robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance retrieval with alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a retrieval-augmented alignment model that consolidates multiple references with dual-weighted attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑