大型语言模型的经济学竞技场
Economics Arena for Large Language Models
AI总结：
本文提出用竞争性博弈动态评估LLM，发现多数模型具理性但未达纳什均衡，GPT-4等在博弈历史可用时收敛更快且胜率更高，并构建了测试理性、策略推理与指令遵循能力的经济学竞技场。
摘要：
大型语言模型（LLMs）已被广泛用作通用智能体的骨干，一些经济学文献表明LLMs能够参与各种类型的经济学博弈。继这些工作之后，为了克服使用静态基准评估LLMs的局限性，我们提出探索竞争性博弈作为LLMs的评估方法，以引入多玩家并使环境动态化。通过改变向基于LLM的玩家揭示的博弈历史，我们发现大多数LLM是理性的，它们会采取能增加自身收益的策略，但未达到纳什均衡所表明的理性程度。此外，当博弈历史可用时，某些类型的LLM（如GPT-4）能更快收敛到NE策略，这表明与其他模型相比具有更高的理性水平。同时，当博弈历史可用时，某些类型的LLM能更频繁地获胜，我们认为胜率反映了对其他玩家策略的推理能力。在所有实验中，我们观察到严格遵循自然语言描述的博弈规则的能力在我们测试的LLM中也存在差异。在这项工作中，我们为LLM研究社区提供了一个经济学竞技场，作为动态模拟来测试LLM的上述能力，即理性、策略推理能力和指令遵循能力。
英文摘要：
Large language models (LLMs) have been extensively used as the backbones for general-purpose agents, and some economics literature suggest that LLMs are capable of playing various types of economics games. Following these works, to overcome the limitation of evaluating LLMs using static benchmarks, we propose to explore competitive games as an evaluation for LLMs to incorporate multi-players and dynamicise the environment. By varying the game history revealed to LLMs-based players, we find that most of LLMs are rational in that they play strategies that can increase their payoffs, but not as rational as indicated by Nash Equilibria (NEs). Moreover, when game history are available, certain types of LLMs, such as GPT-4, can converge faster to the NE strategies, which suggests higher rationality level in comparison to other models. In the meantime, certain types of LLMs can win more often when game history are available, and we argue that the winning rate reflects the reasoning ability with respect to the strategies of other players. Throughout all our experiments, we observe that the ability to strictly follow the game rules described by natural languages also vary among the LLMs we tested. In this work, we provide an economics arena for the LLMs research community as a dynamic simulation to test the above-mentioned abilities of LLMs, i.e. rationality, strategic reasoning ability, and instruction-following capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
