自我评估测试是衡量LLM人格的不可靠指标
Self-Assessment Tests are Unreliable Measures of LLM Personality
AI总结：
本文通过提示敏感性和选项顺序对称性实验证明，自我评估人格测试在衡量LLM人格时不可靠，其分数会随提示和选项顺序显著变化。
摘要：
随着大型语言模型（LLM）能力的不断发展，近期许多研究尝试使用为研究人类行为而创建的心理学工具来量化其行为。其中一个例子是使用为测量人类人格而开发的自我评估人格测试来衡量LLM的“人格”。然而，几乎没有任何工作验证这些测试在LLM上的适用性。在本文中，我们通过两个简单的实验分析了从自我评估人格测试中获得的LLM人格分数的可靠性。我们首先引入了提示敏感性的概念，使用三个语义等价的提示（代表在LLM上实施自我评估测试的三种直观方式）来测量同一个LLM的人格。我们发现，所有三个提示都导致了截然不同的人格分数，在绝大多数场景中，这种差异对于所有人格特质都具有统计学意义。然后，我们引入了用于LLM人格测量的选项顺序对称性概念。由于大多数自我评估测试以多项选择题（MCQ）的形式存在，我们认为分数不仅应对提示模板具有鲁棒性，还应对选项呈现的顺序具有鲁棒性。不出所料，该测试表明自我评估测试分数对选项顺序不具备鲁棒性。这些在ChatGPT和三个不同规模的Llama2模型上进行的简单测试表明，为人类设计的自我评估人格测试是衡量LLM人格的不可靠指标。
英文摘要：
As large language models (LLM) evolve in their capabilities, various recent studies have tried to quantify their behavior using psychological tools created to study human behavior. One such example is the measurement of "personality" of LLMs using self-assessment personality tests developed to measure human personality. Yet almost none of these works verify the applicability of these tests on LLMs. In this paper, we analyze the reliability of LLM personality scores obtained from self-assessment personality tests using two simple experiments. We first introduce the property of prompt sensitivity, where three semantically equivalent prompts representing three intuitive ways of administering self-assessment tests on LLMs are used to measure the personality of the same LLM. We find that all three prompts lead to very different personality scores, a difference that is statistically significant for all traits in a large majority of scenarios. We then introduce the property of option-order symmetry for personality measurement of LLMs. Since most of the self-assessment tests exist in the form of multiple choice question (MCQ) questions, we argue that the scores should also be robust to not just the prompt template but also the order in which the options are presented. This test unsurprisingly reveals that the self-assessment test scores are not robust to the order of the options. These simple tests, done on ChatGPT and three Llama2 models of different sizes, show that self-assessment personality tests created for humans are unreliable measures of personality in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
