你不需要人格测试就知道这些模型不可靠：评估大型语言模型在心理测量工具上的可靠性
You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments
AI总结：
本文通过构建涵盖39种人格测量工具的数据集并设计提示扰动实验，系统评估了17个大型语言模型在心理测量问答中的可靠性，发现简单扰动即显著降低模型表现且否定一致性较差，指出当前提示方法不足以可靠捕捉模型感知。
摘要：
大型语言模型（LLMs）在自然语言理解任务上的多功能性使其在社会科学研究中广受欢迎。为了正确理解LLMs的性质和内在人格，研究人员开展了相关研究，这些研究使用以问题形式呈现的提示来询问LLMs对特定观点的看法。在本研究中，我们谨慎地退后一步，考察当前提示LLMs的形式是否能以一致且稳健的方式引出回答。我们首先构建了一个数据集，其中包含693个问题，涵盖115个人格轴上的39种不同人格测量工具。此外，我们设计了一组包含细微变化的提示，并考察LLMs生成答案的能力，以及提示变化在内容层面的变体下的一致性，例如调换回答选项的顺序或对陈述进行否定。我们在17个不同的LLMs上进行的实验表明，即使是简单的扰动也会显著降低模型的问答能力，并且大多数LLMs的否定一致性较低。我们的结果表明，目前广泛使用的提示做法不足以准确可靠地捕捉模型感知，因此我们讨论了改进这些问题的潜在替代方案。
英文摘要：
The versatility of Large Language Models (LLMs) on natural language understanding tasks has made them popular for research in social sciences. To properly understand the properties and innate personas of LLMs, researchers have performed studies that involve using prompts in the form of questions that ask LLMs about particular opinions. In this study, we take a cautionary step back and examine whether the current format of prompting LLMs elicits responses in a consistent and robust manner. We first construct a dataset that contains 693 questions encompassing 39 different instruments of persona measurement on 115 persona axes. Additionally, we design a set of prompts containing minor variations and examine LLMs' capabilities to generate answers, as well as prompt variations to examine their consistency with respect to content-level variations such as switching the order of response options or negating the statement. Our experiments on 17 different LLMs reveal that even simple perturbations significantly downgrade a model's question-answering ability, and that most LLMs have low negation consistency. Our results suggest that the currently widespread practice of prompting is insufficient to accurately and reliably capture model perceptions, and we therefore discuss potential alternatives to improve these issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
