针对 Web 智能体跨网站泛化与准确性难题,本文提出 Wilbur,通过可微排序、指令合成、智能回溯和自动课程数据训练,在 WebVoyager 上取得最先进结果。
摘要:
在 Web 智能体研究领域,同时实现泛化性与准确性仍是一个具有挑战性的问题。由于网站结构差异很大,现有方法往往会失败。此外,现有的微调和上下文学习技术无法在多个网站之间泛化。我们提出 Wilbur,该方法使用可微排序模型和新颖的指令合成技术,利用来自先前运行的任务演示,以最优方式填充黑盒大语言模型的提示。为了最大化端到端成功率,我们还提出了一种智能回溯机制,能够从错误中学习并恢复。最后,我们表明排序模型可以在来自生成式自动课程的数据上训练;该课程从 LLM 采样具有代表性的目标、运行智能体并自动评估,无需人工标注。Wilbur 在 WebVoyager 基准上取得了最先进结果,总体上比纯文本模型高 8%,在某些网站上最高高出 36%。在同一基准上,尽管 Wilbur 只接收文本输入,其表现仍与一个强大的多模态模型相差不超过 5%;进一步分析表明,大量失败源于操作 Web 时的工程挑战。
英文摘要:
In the realm of web agent research, achieving both generalization and accuracy remains a challenging problem. Due to high variance in website structure, existing approaches often fail. Moreover, existing fine-tuning and in-context learning techniques fail to generalize across multiple websites. We introduce Wilbur, an approach that uses a differentiable ranking model and a novel instruction synthesis technique to optimally populate a black-box large language model's prompt with task demonstrations from previous runs. To maximize end-to-end success rates, we also propose an intelligent backtracking mechanism that learns and recovers from its mistakes. Finally, we show that our ranking model can be trained on data from a generative auto-curriculum which samples representative goals from an LLM, runs the agent, and automatically evaluates it, with no manual annotation. Wilbur achieves state-of-the-art results on the WebVoyager benchmark, beating text-only models by 8% overall, and up to 36% on certain websites. On the same benchmark, Wilbur is within 5% of a strong multi-modal model despite only receiving textual inputs, and further analysis reveals a substantial number of failures are due to engineering challenges of operating the web.
LineRetriever: Planning-Aware Observation Reduction for Web Agents
INSA Lyon(里昂国立应用科学学院);Esker(Esker公司);ServiceNow Research(ServiceNow研究);Mila AI Institute(Mila人工智能研究所);McGill University(麦吉尔大学);Université Claude Bernard Lyon 1(里昂第一大学);LIRIS(LIRIS实验室)
Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity
School of AI Convergence, Soongsil University(顺斯利大学人工智能融合学院);MAUM AI Inc.(MAUM人工智能公司);Department of Intelligent Semiconductors, Soongsil University(顺斯利大学智能半导体系)
StepGuard: Guarding Web Navigation via Single-Step Calibration
School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件工程学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);Xiamen University(厦门大学);Zhejiang Lab(之江实验室);CSIRO(澳大利亚联邦科学与工业研究组织)
TheraMind: A Strategic and Adaptive Agent for Longitudinal Psychological Counseling
Shenzhen University(深圳大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Auckland University of Technology(奥克兰理工大学);University of Science and Technology of China(中国科学技术大学);SKL-IOTSC, CIS, University of Macau(澳门科技研究所、CIS、澳门大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳))
Real-time Monitoring of Economic Shocks using Company Websites
ETH Zurich(苏黎世联邦理工学院);Swiss Economic Institute (KOF)(瑞士经济研究所(KOF));Tinbergen Institute(蒂尔堡大学);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);Centre for Economic Policy Research (CEPR)(经济政策研究中心(CEPR))
WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs
Ping An Bank Co., Ltd.(平安银行股份有限公司);Tsinghua University(清华大学);Central South University(中南大学);Shanghai Yiwo Information Technology Co., Ltd.(上海易沃信息技术有限公司);Kwangwoon University(光云大学)