LLM-Grounder：以大语言模型为智能体的开放词汇3D视觉定位
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent
AI总结：
LLM-Grounder利用大语言模型分解复杂查询并结合视觉定位工具，实现零样本开放词汇3D视觉定位，在ScanRefer上取得领先精度。
摘要：
3D视觉定位是家用机器人的一项关键技能，使其能够根据环境进行导航、操作物体并回答问题。现有方法通常依赖大量标注数据，或在处理复杂语言查询时存在局限性，为此我们提出LLM-Grounder，一种新颖的零样本、开放词汇、基于大语言模型（LLM）的3D视觉定位流程。LLM-Grounder利用LLM将复杂的自然语言查询分解为语义成分，并使用视觉定位工具（如OpenScene或LERF）在3D场景中识别物体。随后，LLM评估候选物体之间的空间关系和常识关系，以做出最终定位决策。我们的方法不需要任何标注训练数据，能够泛化到新的3D场景和任意文本查询。我们在ScanRefer基准上评估LLM-Grounder，展示了最先进的零样本定位准确率。研究结果表明，LLM显著提升了定位能力，尤其是对于复杂语言查询，使LLM-Grounder成为机器人领域3D视觉-语言任务的有效方法。视频和交互式演示可在项目网站https://chat-with-nerf.github.io/上查看。
英文摘要：
3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to decompose complex natural language queries into semantic constituents and employs a visual grounding tool, such as OpenScene or LERF, to identify objects in a 3D scene. The LLM then evaluates the spatial and commonsense relations among the proposed objects to make a final grounding decision. Our method does not require any labeled training data and can generalize to novel 3D scenes and arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and demonstrate state-of-the-art zero-shot grounding accuracy. Our findings indicate that LLMs significantly improve the grounding capability, especially for complex language queries, making LLM-Grounder an effective approach for 3D vision-language tasks in robotics. Videos and interactive demos can be found on the project website https://chat-with-nerf.github.io/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
