Think-Program-reCtify:基于大型语言模型的3D情境推理
Think-Program-reCtify: 3D Situated Reasoning with Large Language Models
AI总结:
提出LLM-TPC框架,通过Think-Program-reCtify循环利用大型语言模型的规划、工具调用和反思能力,解决3D情境推理中的数据稀缺与泛化问题,在SQA3D基准上验证了方法的有效性。
摘要:
本工作致力于解决3D情境推理任务,该任务旨在根据3D环境中的自我中心观测来回答问题。由于需要全面的3D感知和复杂的推理能力,该任务仍具挑战性。针对3D情境推理的端到端监督模型面临数据稀缺和泛化能力不足的问题。受近期利用大型语言模型(LLMs)进行视觉推理成功的启发,我们提出了LLM-TPC,这是一种新颖的框架,通过Think-Program-reCtify循环利用LLMs的规划、工具使用和反思能力。Think阶段首先将组合性问题分解为一系列步骤,随后Program阶段将每个步骤对应到一段代码,并调用精心设计的3D视觉感知模块。最后,若程序执行失败,Rectify阶段将调整计划与代码。在SQA3D基准上的实验与分析证明了我们方法的有效性、可解释性和鲁棒性。代码已公开发布于https://qingrongh.github.io/LLM-TPC/。
英文摘要:
This work addresses the 3D situated reasoning task which aims to answer questions given egocentric observations in a 3D environment. The task remains challenging as it requires comprehensive 3D perception and complex reasoning skills. End-to-end models trained on supervised data for 3D situated reasoning suffer from data scarcity and generalization ability. Inspired by the recent success of leveraging large language models (LLMs) for visual reasoning, we propose LLM-TPC, a novel framework that leverages the planning, tool usage, and reflection capabilities of LLMs through a ThinkProgram-reCtify loop. The Think phase first decomposes the compositional question into a sequence of steps, and then the Program phase grounds each step to a piece of code and calls carefully designed 3D visual perception modules. Finally, the Rectify phase adjusts the plan and code if the program fails to execute. Experiments and analysis on the SQA3D benchmark demonstrate the effectiveness, interpretability and robustness of our method. Our code is publicly available at https://qingrongh.github.io/LLM-TPC/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。