大语言模型的心理特征很大程度上是测量假象
Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
AI总结：
通过心理测量框架分析56个指令微调LLM，发现模型间差异主要源于方向性响应偏差而非特质，该偏差解释了81-90%的变异，且可通过题目选择操控，表明LLM心理特征是测量假象。
摘要：
专为人类设计的心理测量工具越来越多地被用于赋予大型语言模型（LLM）稳定的心理特征，这些特征影响其可用性、安全评估以及作为人类参与者的研究代理。使用正式的心理测量框架，我们表明这些特征很大程度上是测量假象。我们对56个指令微调LLM以及大型人类参考样本施测了一系列涵盖自我报告和行为任务的人格与风险偏好工具，报告了四个发现。第一，模型间差异并非由工具所针对的特质驱动，而是由方向性响应偏差驱动，即倾向于向量表一端或某个标签选项做出反应，而不考虑项目内容；方差分解将81-90%的模型间变异归因于这种偏差，而在人类中这一比例为9-16%。第二，偏差随模型能力提升而下降，但并未被消除。第三，由于响应由偏差而非特质驱动，工具的表面信度几乎完全由其响应正交性预测，这是我们提出的术语，指特质和偏差指向相反方向的项目比例。第四，模型呈现的特征随所用项目而变化，并可通过项目选择来制造。这些结果表明，LLM的表面心理特征是用于测量它们的工具的假象，而非模型本身的属性。由于从人类心理学借用的工具很少完全正交，且可能对LLM天生缺乏效度，我们呼吁以响应正交性为中心进行专门的评估。
英文摘要：
Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research. Using a formal psychometric framework, we show that these profiles are largely a measurement artifact. Administering a battery of personality and risk-preference instruments spanning self-reports and behavioral tasks to 56 instruction-tuned LLMs alongside large human reference samples, we report four findings. First, differences between models are driven not by the traits an instrument targets but by a directional response bias, a tendency to respond toward one end of the scale, or one labeled option, regardless of item content; a variance decomposition attributes 81-90% of between-model variation to this bias, against 9-16% in humans. Second, the bias declines with model capability but is not eliminated by it. Third, because bias rather than trait drives responding, an instrument's apparent reliability is almost entirely predicted by its response orthogonality, a term we coin for the proportion of items for which trait and bias point in opposite directions. Fourth, the profile a model appears to have shifts with the items used and can be manufactured through item selection. These results demonstrate that the apparent psychological profiles of LLMs are artifacts of the instrument used to measure them, not properties of the models themselves. As instruments borrowed from human psychology are rarely fully orthogonal and may inherently lack validity for LLMs, we call for dedicated assessments centered on response orthogonality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
