人类心理测量问卷误判LLM行为
Human Psychometric Questionnaires Mischaracterize LLM Behavior
AI总结：
通过比较LLM在Likert问卷和生成概率上的价值与人格特征，发现问卷存在系统性偏差，提出基于生成概率的评估方法更准确。
摘要：
我们检验了人类心理测量问卷是否可以作为可靠工具来表征和预测LLM在日常用户交互中的行为。我们分析了八个开源LLM，比较了从两种不同方法得出的价值和人格特征：基于既定问卷（PVQ-40/21和BFI-44/10）的Likert自我报告，以及对日常用户查询的价值负载响应的生成概率。两种特征显著不同。在生成概率中，常被引为LLM稳定倾向证据的构念内项目一致性消失了。我们将这一差距归因于既定问卷项目中的显式词汇线索使模型能够识别目标构念并以一致、社会期望的方式响应，而现实用户查询不提供此类线索。此外，人口统计角色提示以与真实人类模式一致的方式改变了模型对人类问卷的响应，但在对现实用户查询的响应生成概率中没有出现此类变化，表明它们在模拟目标人口统计在真实世界用户交互中的行为方面能力有限。总体而言，我们的研究表明，人类心理测量问卷不足以预测LLM行为，并建议基于生成的评估作为更准确的度量。
英文摘要：
We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open-source LLMs by comparing their value and personality profiles derived from two different methods: Likert self-reports on established questionnaires (PVQ-40/21 and BFI-44/10) and generation probabilities over value-laden responses to everyday user queries. The two profiles diverge substantially. Within-construct item consistency, often cited as evidence of stable LLM dispositions, disappears in generation probabilities. We find that established questionnaire items contain explicit lexical cues that allow models to recognize the target construct and respond in alignment-consistent, socially desirable ways, whereas realistic user queries contain far less recognizable cues. In addition, demographic persona prompts shift models' responses to human questionnaires in ways consistent with real human patterns, but no such shifts appear in the generation setting, highlighting that human questionnaires overestimate LLMs' ability to faithfully reproduce expected psychological traits when role-playing demographic personas. Overall, our study indicates that questionnaire scores alone should not be treated as evidence of LLMs' response tendencies in realistic user interactions, and supports generation-probability profiling with ecologically valid items as a complementary behavioral measure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
