arXiv 2606.127302026-06-12cs.AIcs.CLcs.CYcs.LG

重新思考LLMs的心理测量评估：自我报告何时以及为何能预测行为

Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior

Caltech（加州理工学院）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；University of Cambridge（剑桥大学）

Rafal Kocielnik, Pengrui Han, Peiyang Song, Myrl G. Marmarelis, Ramit Debnath, Dean Mobbs, Anima Anandkumar, R. Michael Alvarez

备注：: Accepted as an Oral (Contributed Talk) at the ICML 2026 Workshop on Combining Theory and Benchmarks (CTB)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究对比大五人格与计划行为理论，发现LLMs的自我报告-行为一致性存在选择性：在共享对话中TPB达到人类水平，跨对话仅对锚定于训练的行为保持一致性，且角色提示不能使行为对齐。

摘要：

从低成本心理测量探针预测LLM行为倾向对于安全部署至关重要，但前提是自我报告（SR）能可靠地预测行为。近期研究记录了LLMs中显著的SR-行为分离，但依赖于广泛的人格特质（大五），这些特质即使在人类中也只能弱预测特定行为。此外，对话会话的隔离加上弱上下文匹配使得以下问题悬而未决：LLMs是否真正缺乏一致性，或者检测这种一致性所需的条件是否未满足。我们将大五与计划行为理论（TPB）进行对比，后者测量针对特定行为的意图，并且比广泛特质能更好地预测人类行为。我们在四个行为任务和11个前沿LLM上进行实验，同时改变会话上下文和身份诱导。我们发现SR-行为一致性存在但具有选择性。1) 在共享对话中，计划行为理论达到人类水平的一致性；大五则没有。2) 在跨对话中，一致性仅对锚定于即时提示之外的行为（如由训练塑造的内隐偏见）幸存，而当行为被上下文强烈启动（如谄媚）时则崩溃。3) 角色提示使自我报告在对话间更一致，但并未使行为对齐。这些发现表明，粗糙的人格框架（如大五）可能不是测试部署行为的最佳工具。需要更多任务和特定行为的工具，并且即使这些工具也必须在任务和上下文中进行评估。

英文摘要：

Anticipating LLM behavioral tendencies from low-cost psychometric probes is critical for safe deployment, but only if self-reports (SR) reliably predict behavior. Recent work documented substantial SR-behavior dissociation in LLMs, but relied on broad personality traits (Big 5) that predict specific behaviors weakly, even in humans. Furthermore, the isolation of conversational sessions combined with weak context matching left open whether LLMs truly lack coherence or whether the conditions needed to detect such coherence were not met. We contrast Big 5 with the Theory of Planned Behavior (TPB), which measures intention targeted to a specific behavior and predicts human behavior substantially better than broad traits. We run experiments across four behavioral tasks and 11 frontier LLMs, while also varying session context and identity induction. We find that SR-behavior coherence exists but is selective. 1) Within a shared conversation, the Theory of Planned Behavior reaches human-level coherence; Big 5 does not. 2) Across separate conversations, coherence survives only for behaviors anchored outside the immediate prompt, such as implicit bias shaped by training, and collapses when behavior is strongly primed by context, as with sycophancy. 3) Persona prompting makes self-reports more consistent across conversations, but does not bring behavior into alignment. These findings suggest that coarse personality frameworks, such as Big 5 may not be the best tools for testing deployment behavior. More task- and behavior-specific instruments are needed, and even these must be evaluated across tasks and contexts.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。