arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.14913cs.CLcs.IRcs.LG
收藏

迈向面向开放网络研究智能体的现实长期基准

Towards a Realistic Long-Term Benchmark for Open-Web Research Agents

Peter Mühlbacher, Nikos I. Bosse, Lawrence Phillips

AI总结:

本文提出了一个评估LLM智能体在金融和咨询等现实开放网络研究任务上表现的基准,发现Claude-3.5 Sonnet和o1-preview表现最佳,且能委派子任务的ReAct架构效果最优。

摘要:

我们展示了一个即将推出的基准的初步结果,该基准用于评估LLM智能体在具有经济价值的白领任务上的表现。我们在现实世界“杂乱”的开放网络研究任务上评估智能体,这类任务在金融和咨询领域很常见。借此,我们为LLM智能体评估套件奠定了基础,在该套件中,良好的表现直接对应着巨大的经济和社会影响。我们构建并测试了几种智能体架构,使用了o1-preview、GPT-4o、Claude-3.5 Sonnet、Llama 3.1 (405b)和GPT-4o-mini。平均而言,由Claude-3.5 Sonnet和o1-preview驱动的LLM智能体显著优于使用GPT-4o的智能体,而基于Llama 3.1 (405b)和GPT-4o-mini的智能体则明显落后。在所有LLM中,能够将子任务委派给子智能体的ReAct架构表现最佳。除了定量评估外,我们还通过检查LLM智能体的轨迹并反思其观察结果,定性评估了它们的性能。我们的评估代表了对智能体在真实开放网络上进行具有挑战性、具有经济价值的分析师风格研究能力的首次深入评估。

英文摘要:

We present initial results of a forthcoming benchmark for evaluating LLM agents on white-collar tasks of economic value. We evaluate agents on real-world "messy" open-web research tasks of the type that are routine in finance and consulting. In doing so, we lay the groundwork for an LLM agent evaluation suite where good performance directly corresponds to a large economic and societal impact. We built and tested several agent architectures with o1-preview, GPT-4o, Claude-3.5 Sonnet, Llama 3.1 (405b), and GPT-4o-mini. On average, LLM agents powered by Claude-3.5 Sonnet and o1-preview substantially outperformed agents using GPT-4o, with agents based on Llama 3.1 (405b) and GPT-4o-mini lagging noticeably behind. Across LLMs, a ReAct architecture with the ability to delegate subtasks to subagents performed best. In addition to quantitative evaluations, we qualitatively assessed the performance of the LLM agents by inspecting their traces and reflecting on their observations. Our evaluation represents the first in-depth assessment of agents' abilities to conduct challenging, economically valuable analyst-style research on the real open web.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑