DeliberationBench: 一个用于评估大语言模型对用户观点影响的规范性基准
DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Views
AI总结:
DeliberationBench通过评估大语言模型对用户观点影响的规范性基准,揭示了其在政策讨论中的显著影响及对民主标准的贡献。
摘要:
随着大语言模型(LLMs)作为助手和思想伙伴变得无处不在,有必要描述它们对用户信念的说服影响。然而,一个核心挑战是区分“有益”与“有害”的影响形式,以一种规范性上可辩护且合法的方式。我们提出了DeliberationBench,这是一个用于评估LLM影响的基准,其标准是 deliberative opinion polling的过程。我们在一个预注册的随机实验中展示了我们的方法,其中4,088名美国参与者与六个前沿LLMs讨论了65项政策提案。利用四个先前由Deliberative Democracy Lab进行的 deliberative polls 的意见变化数据,我们发现测试的LLMs的影响在规模上是显著的,并且与经过 deliberation 后的净意见变化正相关,表明这些模型在广义上具有知识上可取的效果。我们进一步探讨了不同主题领域、人口统计数据组和模型之间的差异性影响。我们的框架可以作为评估和监控工具,帮助确保LLMs的影响与民主合法标准保持一致,并保护用户在形成观点方面的自主性。
英文摘要:
As large language models (LLMs) become pervasive as assistants and thought partners, it is important to characterize their persuasive influence on users' beliefs. However, a central challenge is to distinguish "beneficial" from "harmful" forms of influence, in a manner that is normatively defensible and legitimate. We propose DeliberationBench, a benchmark for assessing LLM influence that takes the process of deliberative opinion polling as its standard. We demonstrate our approach in a preregistered randomized experiment in which 4,088 U.S. participants discussed 65 policy proposals with six frontier LLMs. Using opinion change data from four prior Deliberative Polls conducted by the Deliberative Democracy Lab, we find evidence that the tested LLMs' influence is substantial in magnitude and positively associated with the net opinion shifts following deliberation, suggesting that these models exert broadly epistemically desirable effects. We further explore differential influence between topic areas, demographic subgroups, and models. Our framework can function as an evaluation and monitoring tool, helping to ensure that the influence of LLMs remains consistent with democratically legitimate standards, and preserves users' autonomy in forming their views.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。