AssistGPT:一个能够规划、执行、检查和学习的通用多模态助手
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
AI总结:
AssistGPT提出PEIL交错代码与语言推理方法,通过Planner、Inspector和Learner集成LLMs与工具,解决复杂多模态任务,在A-OKVQA和NExT-QA上取得SOTA。
摘要:
近期关于大型语言模型(LLMs)的研究推动了通用自然语言处理AI助手取得显著进展。一些研究进一步探索利用LLMs进行规划并调用模型或API,以解决更通用的多模态用户查询。尽管取得了进展,由于视觉任务的多样性,复杂的基于视觉的任务仍然具有挑战性。这种多样性体现在两个方面:1)推理路径。对于许多实际应用,仅通过检查查询本身很难准确分解查询。通常需要基于具体的视觉内容和每一步的结果进行规划。2)灵活的输入和中间结果。对于真实场景,输入形式可能很灵活,不仅涉及单个图像或视频,还涉及视频和图像的混合,例如带有参考视频的用户视角图像。此外,复杂的推理过程也会产生多样的多模态中间结果,例如视频叙述、分割的视频片段等。为了解决此类通用情况,我们提出了一种多模态AI助手AssistGPT,采用一种名为规划、执行、检查和学习(PEIL)的交错代码与语言推理方法,将LLMs与各种工具集成。具体而言,Planner能够使用自然语言,根据当前推理进度规划Executor中接下来应使用哪个工具。Inspector是一个高效的内存管理器,协助Planner将适当的视觉信息输入到特定工具中。最后,由于整个推理过程复杂且灵活,设计了一个Learner使模型能够自主探索并发现最优解。我们在A-OKVQA和NExT-QA基准上进行了实验,取得了最先进的结果。此外,展示案例证明了我们的系统处理远比基准中问题复杂的问题的能力。
英文摘要:
Recent research on Large Language Models (LLMs) has led to remarkable advancements in general NLP AI assistants. Some studies have further explored the use of LLMs for planning and invoking models or APIs to address more general multi-modal user queries. Despite this progress, complex visual-based tasks still remain challenging due to the diverse nature of visual tasks. This diversity is reflected in two aspects: 1) Reasoning paths. For many real-life applications, it is hard to accurately decompose a query simply by examining the query itself. Planning based on the specific visual content and the results of each step is usually required. 2) Flexible inputs and intermediate results. Input forms could be flexible for in-the-wild cases, and involves not only a single image or video but a mixture of videos and images, e.g., a user-view image with some reference videos. Besides, a complex reasoning process will also generate diverse multimodal intermediate results, e.g., video narrations, segmented video clips, etc. To address such general cases, we propose a multi-modal AI assistant, AssistGPT, with an interleaved code and language reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate LLMs with various tools. Specifically, the Planner is capable of using natural language to plan which tool in Executor should do next based on the current reasoning progress. Inspector is an efficient memory manager to assist the Planner to feed proper visual information into a specific tool. Finally, since the entire reasoning process is complex and flexible, a Learner is designed to enable the model to autonomously explore and discover the optimal solution. We conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving state-of-the-art results. Moreover, showcases demonstrate the ability of our system to handle questions far more complex than those found in the benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。