arXivDaily arXiv每日学术速递 周一至周五更新
​通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2302.07267cs.HCcs.AIcs.CL

标准大型语言模型中思维多样性的减弱

Diminished Diversity-of-Thought in a Standard Large Language Model

MIT（麻省理工学院）；London School of Economics（伦敦政治经济学院）；CVS Health（CVS健康公司）

Peter S. Park, Philipp Schoenegger, Chongyang Zhu

备注：
67 pages (42-page main text, 25-page SI); 12 visualizations (four tables and three figures in the main text, five figures in the SI); additional exploratory follow-up study varied the demographic details preceding the prompt; preregistered OSF database is available at https://osf.io/dzp8t/
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究测试GPT3.5模拟社会科学人类参与者的能力，发现其因“正确答案”效应导致回答缺乏多样性，对LLMs替代人类参与者的有效性提出质疑，并担忧AI主导的未来思维多样性将减弱。

摘要：

我们测试了大型语言模型（LLMs）是否可用于模拟社会科学研究中的人类参与者。为此，我们使用OpenAI的text-davinci-003模型（俗称GPT3.5）对Many Labs 2复制项目中的14项研究进行了复制。根据我们的预注册分析，在我们能够分析的8项研究中，我们的GPT样本复制了37.5%的原始结果和37.5%的Many Labs 2结果。然而，由于一种我们称之为“正确答案”效应的意外现象，我们无法分析剩余的6项研究。GPT3.5的不同运行在回答探究政治取向、经济偏好、判断和道德哲学的细微问题时，其回答的变化为零或接近零：即给出了所谓的“正确答案”。在一项探索性后续研究中，我们发现“正确答案”对于更改提示前的人口统计细节具有鲁棒性。在另一项研究中，我们发现大多数（但非全部）“正确答案”对于更改答案选项的顺序具有鲁棒性。我们最引人注目的发现之一出现在对Moral Foundations Theory调查结果的复制中，我们发现GPT3.5在99.6%的情况下自认为政治保守派，而在逆序条件下，在99.3%的情况下自认为自由派。然而，自我报告的“GPT保守派”和“GPT自由派”都表现出右倾的道德基础。我们的结果对在社会科学中使用LLMs作为人类参与者的一般替代品的有效性产生了怀疑。我们的结果也引起了人们对假设的AI主导的未来可能会面临思维多样性减弱的担忧。

英文摘要：

We test whether Large Language Models (LLMs) can be used to simulate human participants in social-science studies. To do this, we run replications of 14 studies from the Many Labs 2 replication project with OpenAI's text-davinci-003 model, colloquially known as GPT3.5. Based on our pre-registered analyses, we find that among the eight studies we could analyse, our GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results. However, we were unable to analyse the remaining six studies due to an unexpected phenomenon we call the "correct answer" effect. Different runs of GPT3.5 answered nuanced questions probing political orientation, economic preference, judgement, and moral philosophy with zero or near-zero variation in responses: with the supposedly "correct answer." In one exploratory follow-up study, we found that a "correct answer" was robust to changing the demographic details that precede the prompt. In another, we found that most but not all "correct answers" were robust to changing the order of answer choices. One of our most striking findings occurred in our replication of the Moral Foundations Theory survey results, where we found GPT3.5 identifying as a political conservative in 99.6% of the cases, and as a liberal in 99.3% of the cases in the reverse-order condition. However, both self-reported 'GPT conservatives' and 'GPT liberals' showed right-leaning moral foundations. Our results cast doubts on the validity of using LLMs as a general replacement for human participants in the social sciences. Our results also raise concerns that a hypothetical AI-led future may be subject to a diminished diversity-of-thought.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑