AI 玩商业游戏：在动态模拟中对大型语言模型的管理决策能力进行基准测试
AI Playing Business Games: Benchmarking Large Language Models on Managerial Decision-Making in Dynamic Simulations
AI总结：
本研究提出一种用于LLM基准测试的可复现开放获取管理模拟器，评估Gemini等五个LLMs在动态商业模拟中的多步骤战略决策能力，为长期决策评估提供新途径。
摘要：
大型语言模型（LLMs）的快速发展引发了人们对其增强或自动化管理职能潜力的浓厚兴趣。AI基准测试的最新趋势之一是评估大型语言模型（LLMs）在更长时间范围内的表现。尽管LLMs在涉及自然语言和模式识别的任务中表现出色，但它们在多步骤、战略性商业决策方面的能力在很大程度上仍未被探索。少数研究表明，结果可能与短期任务的基准测试不同，例如Vending-Bench所揭示的。与此同时，缺乏针对长期连贯性的替代基准测试。本研究分析了一种使用商业游戏进行商业决策的新型基准测试。该研究通过向研究社区提出一个可复现、开放获取的管理模拟器，用于LLM基准测试，为近期AI文献做出了贡献。这个新型框架用于评估五个可通过免费在线接口获取的领先LLMs的表现：Gemini、ChatGPT、Meta AI、Mistral AI和Grok。LLMs为一个模拟的零售公司做出决策。动态的、按月进行的管理模拟以电子表格模型的形式透明地作为实验环境提供。在十二个月中的每个月，LLMs都会收到一个结构化提示，其中包含前一时期的完整商业报告，并被要求做出关键战略决策：定价、订单规模、营销预算、招聘、裁员、贷款、培训费用、研发费用、销售预测、收入预测。该方法旨在通过定量指标（利润、收入和市场份额）以及其他关键绩效指标（KPIs）来比较LLMs。对LLMs的决策在战略连贯性、对市场变化的适应性以及为决策提供的理由方面进行了分析。这种方法使得能够超越简单的性能指标来评估长期决策能力。
英文摘要：
The rapid advancement of LLMs sparked significant interest in their potential to augment or automate managerial functions. One of the most recent trends in AI benchmarking is performance of Large Language Models (LLMs) over longer time horizons. While LLMs excel at tasks involving natural language and pattern recognition, their capabilities in multi-step, strategic business decision-making remain largely unexplored. Few studies demonstrated how results can be different from benchmarks in short-term tasks, as Vending-Bench revealed. Meanwhile, there is a shortage of alternative benchmarks for long-term coherence. This research analyses a novel benchmark using a business game for the decision making in business. The research contributes to the recent literature on AI by proposing a reproducible, open-access management simulator to the research community for LLM benchmarking. This novel framework is used for evaluating the performance of five leading LLMs available in free online interface: Gemini, ChatGPT, Meta AI, Mistral AI, and Grok. LLM makes decisions for a simulated retail company. A dynamic, month-by-month management simulation provides transparently in spreadsheet model as experimental environment. In each of twelve months, the LLMs are provided with a structured prompt containing a full business report from the previous period and are tasked with making key strategic decisions: pricing, order size, marketing budget, hiring, dismissal, loans, training expense, R&D expense, sales forecast, income forecast The methodology is designed to compare the LLMs on quantitative metrics: profit, revenue, and market share, and other KPIs. LLM decisions are analyzed in their strategic coherence, adaptability to market changes, and the rationale provided for their decisions. This approach allows to move beyond simple performance metrics for assessment of the long-term decision-making.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
