arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.16356cs.LGcs.AI
收藏

评估大型语言模型中的二元决策偏差:对公平的基于智能体的金融模拟的影响

Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulations

UNSW Sydney(新南威尔士大学悉尼分校)

Alicia Vidler, Toby Walsh

AI总结:

本文评估了三种GPT模型在二元决策中的偏差,发现模型子版本和采样方法显著影响输出分布,强调将LLM谨慎整合到基于智能体的金融市场模型中的重要性。

摘要:

大型语言模型(LLMs)正越来越多地被用于在基于智能体的金融市场模型(ABMs)中模拟类人决策。随着模型变得更加强大和易于访问,研究人员现在可以将单个 LLM 决策纳入 ABM 环境。然而,这种整合可能会引入需要仔细评估的固有偏差。在本文中,我们使用两种模型采样方法:一次性(one-shot)和少样本(few-shot)API 查询,测试了三种最先进的 GPT 模型的偏差。我们观察到特定模型和模型子版本之间的输出分布存在显著差异,其中 GPT-4o-Mini-2024-07-18 表现出明显更好的性能(32-43% 的肯定响应),而 GPT-4-0125-preview 则表现出极端偏差(98-99% 的肯定响应)。我们表明采样方法和模型子版本对结果有显著影响:重复的独立 API 调用与单次调用内的批量采样产生不同的分布。虽然目前没有 GPT 模型能够在一次性测试中同时实现均匀分布和马尔可夫性质,但少样本采样在某些条件下可以接近均匀分布。我们探讨了 Temperature 参数,提供了定义和比较结果。我们进一步将我们的结果与真正的随机二元序列进行比较,并专门测试了人类常见的负近因偏差——发现 LLMs 在这一方面“击败”人类的能力参差不齐。这些发现强调了将 LLM 谨慎整合到金融市场 ABMs 及更广泛领域中的至关重要性。

英文摘要:

Large Language Models (LLMs) are increasingly being used to simulate human-like decision making in agent-based financial market models (ABMs). As models become more powerful and accessible, researchers can now incorporate individual LLM decisions into ABM environments. However, integration may introduce inherent biases that need careful evaluation. In this paper we test three state-of-the-art GPT models for bias using two model sampling approaches: one-shot and few-shot API queries. We observe significant variations in distributions of outputs between specific models, and model sub versions, with GPT-4o-Mini-2024-07-18 showing notably better performance (32-43% yes responses) compared to GPT-4-0125-preview's extreme bias (98-99% yes responses). We show that sampling methods and model sub-versions significantly impact results: repeated independent API calls produce different distributions compared to batch sampling within a single call. While no current GPT model can simultaneously achieve a uniform distribution and Markovian properties in one-shot testing, few-shot sampling can approach uniform distributions under certain conditions. We explore the Temperature parameter, providing a definition and comparative results. We further compare our results to true random binary series and test specifically for the common human bias of Negative Recency - finding LLMs have a mixed ability to 'beat' humans in this one regard. These findings emphasise the critical importance of careful LLM integration into ABMs for financial markets and more broadly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑