3D-AffordanceLLM：利用大语言模型进行3D世界中的开放词汇可供性检测
3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds
AI总结：
本文将3D可供性检测重新表述为指令推理可供性分割任务，提出3D-AffordanceLLM框架，通过引入大语言模型和定制解码器实现开放世界推理可供性检测，并采用多阶段训练策略，在开放词汇可供性检测任务上实现了约8%的mIoU提升。
摘要：
3D可供性检测是一个具有挑战性的问题，在各种机器人任务中有着广泛的应用。现有方法通常将检测范式表述为基于标签的语义分割任务。这种范式依赖于预定义的标签，并且缺乏理解复杂自然语言的能力，导致在开放世界场景中的泛化能力有限。为了解决这些限制，我们将传统的可供性检测范式重新表述为指令推理可供性分割（IRAS）任务。该任务旨在给定查询推理文本时输出可供性掩码区域，从而避免了输入标签的固定类别。我们相应地提出了3D-AffordanceLLM（3D-ADLLM），这是一个专为3D开放场景中的推理可供性检测设计的框架。具体而言，3D-ADLLM将大语言模型（LLMs）引入3D可供性感知，并配以专门设计的解码器用于生成可供性掩码，从而实现开放世界推理可供性检测。此外，鉴于用于训练大型模型的3D可供性数据集的稀缺性，我们寻求从通用分割数据中提取知识并将其迁移到可供性检测中。因此，我们提出了一种多阶段训练策略，该策略始于一个新颖的预训练任务，即引用对象部位分割（ROPS）。这一阶段旨在赋予模型在对象部位层面的通用识别和分割能力。随后通过对IRAS任务进行微调，3D-ADLLM获得了可供性检测的推理能力。总而言之，3D-ADLLM利用了LLMs丰富的世界知识和人-物交互推理能力，在开放词汇可供性检测任务上实现了约8%的mIoU提升。
英文摘要：
3D Affordance detection is a challenging problem with broad applications on various robotic tasks. Existing methods typically formulate the detection paradigm as a label-based semantic segmentation task. This paradigm relies on predefined labels and lacks the ability to comprehend complex natural language, resulting in limited generalization in open-world scene. To address these limitations, we reformulate the traditional affordance detection paradigm into \textit{Instruction Reasoning Affordance Segmentation} (IRAS) task. This task is designed to output a affordance mask region given a query reasoning text, which avoids fixed categories of input labels. We accordingly propose the \textit{3D-AffordanceLLM} (3D-ADLLM), a framework designed for reasoning affordance detection in 3D open-scene. Specifically, 3D-ADLLM introduces large language models (LLMs) to 3D affordance perception with a custom-designed decoder for generating affordance masks, thus achieving open-world reasoning affordance detection. In addition, given the scarcity of 3D affordance datasets for training large models, we seek to extract knowledge from general segmentation data and transfer it to affordance detection. Thus, we propose a multi-stage training strategy that begins with a novel pre-training task, i.e., \textit{Referring Object Part Segmentation}~(ROPS). This stage is designed to equip the model with general recognition and segmentation capabilities at the object-part level. Then followed by fine-tuning with the IRAS task, 3D-ADLLM obtains the reasoning ability for affordance detection. In summary, 3D-ADLLM leverages the rich world knowledge and human-object interaction reasoning ability of LLMs, achieving approximately an 8\% improvement in mIoU on open-vocabulary affordance detection tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
