In this paper, we propose an autonomous information seeking visual question answering framework, AVIS. Our method leverages a Large Language Model (LLM) to dynamically strategize the utilization of external tools and to investigate their outputs, thereby acquiring the indispensable knowledge needed to provide answers to the posed questions. Responding to visual questions that necessitate external knowledge, such as "What event is commemorated by the building depicted in this image?", is a complex task. This task presents a combinatorial search space that demands a sequence of actions, including invoking APIs, analyzing their responses, and making informed decisions. We conduct a user study to collect a variety of instances of human decision-making when faced with this task. This data is then used to design a system comprised of three components: an LLM-powered planner that dynamically determines which tool to use next, an LLM-powered reasoner that analyzes and extracts key information from the tool outputs, and a working memory component that retains the acquired information throughout the process. The collected user behavior serves as a guide for our system in two key ways. First, we create a transition graph by analyzing the sequence of decisions made by users. This graph delineates distinct states and confines the set of actions available at each state. Second, we use examples of user decision-making to provide our LLM-powered planner and reasoner with relevant contextual instances, enhancing their capacity to make informed decisions. We show that AVIS achieves state-of-the-art results on knowledge-intensive visual question answering benchmarks such as Infoseek and OK-VQA.
Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems
National University of Singapore(新加坡国立大学);Polytechnic University of Bari(巴里理工大学);Renmin University of China(中国人民大学);University of Science and Technology of China(中国科学技术大学)
QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model
The Hong Kong University of Science and Technology(香港科技大学);IDEA Research(IDEA研究院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Supervising the search process produces reliable and generalizable information-seeking agents
Department of Computer Science, University of Virginia, USA(弗吉尼亚大学计算机科学系);National Library of Medicine, National Institutes of Health, USA(美国国立卫生研究院国家医学图书馆);Department of Computer Science, University of Illinois Urbana–Champaign, USA(伊利诺伊大学厄巴纳-香槟分校计算机科学系);Medical Oncology, Dana–Farber Cancer Institute, USA(达纳-法伯癌症研究所医学肿瘤科);Surgery, University of Alabama at Birmingham, USA(阿拉巴马大学伯明翰分校外科系);Department of Neurology, Yale School of Medicine, USA(耶鲁医学院神经病学系)
Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support
Atman Labs(阿特曼实验室);Oxford University Hospitals(牛津大学医院);University of Oxford(牛津大学);NIHR Oxford Biomedical Research Centre(英国国立卫生研究院牛津生物医学研究中心);Imperial College London(伦敦帝国理工学院);Technical University of Munich(慕尼黑工业大学);Massachusetts General Hospital(麻省总医院);Mass General Brigham(麻省总医院布里格姆医疗系统);Harvard Medical School(哈佛医学院);Barts Health NHS Trust(巴特保健国民保健信托基金会);the University of Texas at Austin(德克萨斯大学奥斯汀分校)
From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models
Salesforce AI Research(Salesforce AI研究院);Intuit(Intuit公司);Vanderbilt University(范德比大学);University of California, Davis(加州大学戴维斯分校);Vanderbilt University Medical Center(范德比大学医学中心)
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Department of Computer Science(计算机科学系);University of Oxford(牛津大学);Institute of Intelligent Systems and Robotics(智能系统与机器人研究所);Sorbonne Université(索邦大学);The Leverhulme Centre for the Future of Intelligence(未来智能中心);University of Cambridge(剑桥大学);Independent Researcher(独立研究者);Department of Computer Science and Technology(计算机科学与技术系);Department of Engineering(工程系)
Human Values Matter: Investigating How Misalignment Shapes Collective Behaviors in LLM Agent Communities
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Microsoft Research Asia(微软亚洲研究院);Department of Sociology, Zhejiang University(浙江大学社会学系)
From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Tsinghua University(清华大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Peking University(北京大学);University of California, Los Angeles(加州大学洛杉矶分校);Salesforce AI Research(Salesforce AI研究);Zhejiang University of Technology(浙江工业大学);The Hong Kong University of Science and Technology(香港科学与技术大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Amazon(亚马逊);University of California, San Diego(加州大学圣地亚哥分校);The Chinese University of Hong Kong(香港中文大学)
Towards Knowledgeable Deep Research: Framework and Benchmark
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全国家重点实验室);National University of Singapore(新加坡国立大学);University of Chinese Academy of Sciences(中国科学院大学)