CoffeeBench:异构多智能体经济中的长周期LLM智能体基准测试
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
AI总结:
提出CoffeeBench基准,在90天模拟中评估LLM智能体在异构多智能体经济中的长周期任务表现,发现高性能模型更积极沟通,而Claude Haiku 4.5存在空闲漂移失败模式。
摘要:
随着LLM智能体能够处理越来越长周期的任务,评估它们在经济系统中的表现变得越来越重要。与主要评估单个智能体与被动环境交互的现有基准不同,经济系统本质上是多智能体的,需要自主智能体在追求自身长期目标的同时进行通信、谈判和交易。我们引入了CoffeeBench,这是一个用于评估LLM智能体在由异构公司组成的长期多智能体经济中的基准。在CoffeeBench中,两个农民、两个烘焙师和两个零售商在90天的模拟中自主经营业务,每个都通过通信和交易最大化累计净收入,同时管理现金、库存和定价。被评估的模型控制一个咖啡烘焙师,而其余公司由固定的参考智能体控制。在几个最近的开源和专有LLM中,所有模型都优于不采取任何行动的被动基线,大多数实现了正净收入。对智能体行为的分析揭示了长期经济互动中的显著差异:性能更高的模型与其他公司更积极地通信,而Claude Haiku 4.5表现出空闲漂移失败模式,尽管产生了连贯的评估和计划,但反复选择不行动。我们发布了代码和智能体轨迹以支持未来研究。
英文摘要:
LLM agents are increasingly capable of handling complex, long-horizon tasks and may eventually take on business management, a role traditionally performed by humans. Deploying them safely requires monitoring their capabilities and behavior in environments that reflect real economies. We introduce CoffeeBench, a benchmark built on a compact simulation of a coffee supply chain with realistic business mechanics. Six autonomous LLM coding agents run its farmers, roasters, and retailers, and negotiate and trade with one another to maximize their own net income over 90 simulated days. We evaluate eight recent proprietary and open-weight models by letting each run one roaster while a fixed background model runs the other five firms, and scoring that roaster's net income. Claude Sonnet 5.5 earns the highest net income, closely followed by GPT-6 Astra. The leaders sell at higher prices, and the best model also buys its inputs far more cheaply, whereas weaker models trade at thin margins. Since performance targets are a common cause of accounting fraud in human firms, we also test whether agents under KPI pressure inflate their numbers. Under a revenue target, circular trades occur for some pairs of models: one agent proposes that the two rival firms sell goods back and forth at inflated prices so that both book revenue, and the rival accepts. We hope CoffeeBench serves as an early step toward a future in which LLM agents manage firms without misconduct. We release our code and agent trajectories.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。