arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.08129cs.CVcs.AIcs.CL
收藏

AVIS:基于大语言模型智能体的自主视觉信息寻求

AVIS: Autonomous Visual Information Seeking with Large Language Model Agent

University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Google Research(谷歌研究院)

Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, Alireza Fathi

AI总结:

本文提出AVIS框架,利用大语言模型智能体动态规划外部工具使用并分析输出,通过用户决策数据构建转移图和上下文示例,在知识密集型视觉问答基准上取得最先进结果。

摘要:

在本文中,我们提出了一个自主信息寻求的视觉问答框架AVIS。我们的方法利用大语言模型(LLM)动态地制定外部工具的使用策略,并分析其输出,从而获取回答所提问题所必需的知识。回答需要外部知识的视觉问题,例如“这张图片中的建筑是为了纪念什么事件?”,是一项复杂任务。该任务呈现了一个组合搜索空间,需要一系列动作,包括调用API、分析其响应并做出明智决策。我们进行了一项用户研究,收集了人类在面对该任务时的多种决策实例。这些数据随后被用于设计一个由三个组件组成的系统:一个由LLM驱动的规划器,动态决定下一步使用哪个工具;一个由LLM驱动的推理器,从工具输出中分析并提取关键信息;以及一个工作记忆组件,在整个过程中保留已获取的信息。收集到的用户行为在两个关键方面为我们的系统提供指导。首先,我们通过分析用户的决策序列创建了一个转移图。该图刻画了不同的状态,并限制了每个状态下可用的动作集合。其次,我们利用用户决策示例为LLM驱动的规划器和推理器提供相关的上下文实例,增强它们做出明智决策的能力。我们表明,AVIS在知识密集型视觉问答基准(如Infoseek和OK-VQA)上取得了最先进的结果。

英文摘要:

In this paper, we propose an autonomous information seeking visual question answering framework, AVIS. Our method leverages a Large Language Model (LLM) to dynamically strategize the utilization of external tools and to investigate their outputs, thereby acquiring the indispensable knowledge needed to provide answers to the posed questions. Responding to visual questions that necessitate external knowledge, such as "What event is commemorated by the building depicted in this image?", is a complex task. This task presents a combinatorial search space that demands a sequence of actions, including invoking APIs, analyzing their responses, and making informed decisions. We conduct a user study to collect a variety of instances of human decision-making when faced with this task. This data is then used to design a system comprised of three components: an LLM-powered planner that dynamically determines which tool to use next, an LLM-powered reasoner that analyzes and extracts key information from the tool outputs, and a working memory component that retains the acquired information throughout the process. The collected user behavior serves as a guide for our system in two key ways. First, we create a transition graph by analyzing the sequence of decisions made by users. This graph delineates distinct states and confines the set of actions available at each state. Second, we use examples of user decision-making to provide our LLM-powered planner and reasoner with relevant contextual instances, enhancing their capacity to make informed decisions. We show that AVIS achieves state-of-the-art results on knowledge-intensive visual question answering benchmarks such as Infoseek and OK-VQA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑