arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13209q-fin.GNcs.AI
收藏

LemonadeBench:评估大语言模型在简单市场中的经济直觉

LemonadeBench: Evaluating the Economic Intuition of Large Language Models in Simple Markets

Aidan Vyas

AI总结:

LemonadeBench通过模拟柠檬摊业务评估大语言模型的经济直觉、长期规划和不确定性下的决策能力,揭示模型在局部优化上的表现与全局优化的差距。

摘要:

我们介绍了LemonadeBench v0.5,这是一个最小化基准,用于通过模拟柠檬摊业务来评估大语言模型(LLMs)的经济直觉、长期规划和在不确定性下的决策能力。模型必须管理有保质期的商品库存,设定价格,选择营业时间,并在30天内最大化利润——这些任务是任何小企业主每天都会面对的。所有模型都通过实现盈利展示了有意义的经济代理能力,性能随着复杂度的提高而大幅提升——从基础模型获得最小利润到前沿模型达到理论最优的70%,这一表现超过了10倍的提升。然而,我们对业务效率在六个维度上的分解揭示了一种一致的模式:模型实现了局部而非全局优化,在某些领域表现出色,同时在其他领域却表现出令人惊讶的盲点。

英文摘要:

We introduce LemonadeBench v0.5, a minimal benchmark for evaluating economic intuition, long-term planning, and decision-making under uncertainty in large language models (LLMs) through a simulated lemonade stand business. Models must manage inventory with expiring goods, set prices, choose operating hours, and maximize profit over a 30-day period-tasks that any small business owner faces daily. All models demonstrate meaningful economic agency by achieving profitability, with performance scaling dramatically by sophistication-from basic models earning minimal profits to frontier models capturing 70% of theoretical optimal, a greater than 10x improvement. Yet our decomposition of business efficiency across six dimensions reveals a consistent pattern: models achieve local rather than global optimization, excelling in select areas while exhibiting surprising blind spots elsewhere.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑