AffordanceLLM：从视觉语言模型中实现可供性 grounding
AffordanceLLM: Grounding Affordance from Vision Language Models
AI总结：
针对可供性grounding任务泛化性不足的问题，提出AffordanceLLM模型，借助预训练视觉语言模型的丰富知识提升泛化能力，在AGD20K基准上取得显著性能提升，还能处理训练中未见的物体与动作的可供性定位。
摘要：
可供性grounding是指寻找物体上可与之交互的区域的任务。这是一项基础但极具挑战性的任务，因为成功的解决方案需要从多个方面全面理解场景，包括检测、定位和识别物体及其部件，理解场景的地理空间配置/布局，掌握3D形状与物理特性，以及明确物体与人类的功能和潜在交互方式。这类知识大多是隐性的，超出了有限训练集带监督标签的图像内容范围。本文中，我们尝试利用预训练大规模视觉语言模型中丰富的世界知识、抽象知识以及人-物交互知识，提升当前可供性grounding的泛化能力。在AGD20K基准测试中，我们提出的模型在野外物体可供性grounding任务上的性能显著优于对比方法。我们进一步证明，该模型可以为来自随机互联网图像的物体进行可供性grounding，即便这些物体和动作在训练期间从未见过。项目主页：https://jasonqsy.github.io/AffordanceLLM/
英文摘要：
Affordance grounding refers to the task of finding the area of an object with which one can interact. It is a fundamental but challenging task, as a successful solution requires the comprehensive understanding of a scene in multiple aspects including detection, localization, and recognition of objects with their parts, of geo-spatial configuration/layout of the scene, of 3D shapes and physics, as well as of the functionality and potential interaction of the objects and humans. Much of the knowledge is hidden and beyond the image content with the supervised labels from a limited training set. In this paper, we make an attempt to improve the generalization capability of the current affordance grounding by taking the advantage of the rich world, abstract, and human-object-interaction knowledge from pretrained large-scale vision language models. Under the AGD20K benchmark, our proposed model demonstrates a significant performance gain over the competing methods for in-the-wild object affordance grounding. We further demonstrate it can ground affordance for objects from random Internet images, even if both objects and actions are unseen during training. Project site: https://jasonqsy.github.io/AffordanceLLM/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
