STEER:评估大语言模型的经济理性
STEER: Assessing the Economic Rationality of Large Language Models
AI总结:
本文提出STEER方法论,通过分类经济理性要素、构建基准分布和报告卡,对14种大语言模型进行大规模实验,评估其作为决策智能体的经济理性表现,并分析模型规模的影响。
摘要:
人们越来越有兴趣将大语言模型(LLM)用作决策“智能体”。这样做涉及许多自由度:应该使用哪个模型;应该如何提示它;是否应该要求它进行内省、进行思维链推理等?解决这些问题——以及更广泛地确定LLM智能体是否足够可靠以值得信赖——需要一种评估此类智能体经济理性的方法论。在本文中,我们提供了一种这样的方法论。我们首先调查关于理性决策的经济学文献,对智能体应该表现出的大量细粒度“要素”及其之间的依赖关系进行分类。然后,我们提出一个基准分布,用于定量评分LLM在这些要素上的表现,并结合用户提供的评分标准,生成一份“STEER报告卡”。最后,我们描述了一项涉及14种不同LLM的大规模实证实验的结果,该实验既描述了当前最先进的技术水平,也描述了不同模型规模对模型表现出理性行为能力的影响。
英文摘要:
There is increasing interest in using LLMs as decision-making "agents." Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these questions -- and more broadly, determining whether an LLM agent is reliable enough to be trusted -- requires a methodology for assessing such an agent's economic rationality. In this paper, we provide one. We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them. We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements and, combined with a user-provided rubric, produces a "STEER report card." Finally, we describe the results of a large-scale empirical experiment with 14 different LLMs, characterizing the both current state of the art and the impact of different model sizes on models' ability to exhibit rational behavior.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。