arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.15840cs.AI
收藏

Vending-Bench:一个面向自主智能体长期一致性的基准测试

Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

Andon Labs(安登实验室)

Axel Backlund, Lukas Petersson

AI总结:

本文提出测试自主智能体长期决策一致性的基准Vending-Bench,通过模拟自动售货机运营场景发现多款LLM长期表现差异大且故障与内存限制无关,还可测试模型资本获取能力,助力为更强AI系统做准备。

摘要:

尽管大语言模型(LLM)在孤立的短期任务中能展现出出色的能力,但在更长的时间跨度内,它们往往无法维持连贯的表现。本文提出了Vending-Bench,这是一个专门用于测试基于LLM的智能体管理简单长期商业场景(即运营自动售货机)能力的模拟环境。智能体必须平衡库存、下达订单、设定价格并处理日常费用——这些任务各自都很简单,但在长期运行(每次运行超过2000万token)的情况下,它们共同对LLM维持连贯决策的能力构成了考验。我们的实验显示,多款LLM的表现差异很大:Claude 3.5 Sonnet和o3-mini在大多数运行中能很好地管理机器并实现盈利,但所有模型都存在运行失控的情况,原因包括误解配送时间表、忘记订单,或是陷入几乎无法从中恢复的无关“崩溃”循环。我们发现,故障与模型上下文窗口被填满的时间点之间没有明显的相关性,这表明这些故障并非源于内存限制。除了凸显长期运行下表现的高方差外,Vending-Bench还测试了模型获取资本的能力,这是许多假设的危险AI场景中的一项必要能力。我们希望该基准能助力为更强AI系统的到来做好准备。

英文摘要:

While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑