一个原生LLM的心理测量工具不能预测LLM行为：来自25个模型的证据
An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models
AI总结：
通过探索性因子分析从LLM行为中构建心理测量工具，发现LLM的自我报告与观察行为无关，揭示自我报告与人类判断之间的混淆因素。
摘要：
大型语言模型（LLM）在人格量表上产生稳定的自我报告，但这些自我报告并不能预测观察到的行为。这一差距是反映了LLM与人类特质结构之间的不匹配，还是LLM自我报告本身的更深层属性，此前尚未解决。我们构建了第一个心理测量工具，其结构通过探索性因子分析（EFA）从LLM行为能力中自下而上地推导出来。我们对来自17个模型家族的25个LLM施测了300个项目（240个直接李克特+60个基于场景），涵盖12个候选行为维度，每个项目施测30次。EFA产生了一个5因子结构——响应性、顺从性、大胆性、谨慎性和健谈性——具有极好的分半信度（所有Tucker φ ≥ .957）和内部一致性（所有α ≥ .930）。为了测试预测效度，我们收集了由151名人类评分者和一个三人LLM评审团评分的2500个开放式行为样本。人类和评审团评分一致（r̄ = .51），但两者均不跟踪自我报告：自我报告-人类r̄ = -.01，自我报告-评审团r̄ = .13，且没有因子水平的自我报告-人类置信区间排除零。在响应性上，自我报告与LLM评审团相关（r = .53），但与人类不相关（r = .04），尽管人类和评审团一致（r = .59）——这表明自我报告项目和LLM评审团共享人类观察者未捕捉到的方差，这是一个在集成内部可靠性检查中不可见的混淆因素。我们将该工具作为诊断探针发布，用于检测对齐塑造的自我描述，并作为LLM作为评审团流程的具体风险因素。
英文摘要：
Do large language models' (LLMs') answers to self-report questionnaires predict how they behave? Prior work finds they do not, but it uses human personality inventories, so the gap could reflect borrowed human constructs rather than LLM self-report itself. We test this with a self-report instrument built from LLM-specific behaviors (e.g., over-refusal, unsolicited disclaimers) whose structure is derived bottom-up. Administering 300 items 30 times to 25 LLMs from 17 developers yields five replicable, reliable factors (Tucker $ϕ\geq .957$, $α\geq .930$). We compare these self-reports with 2,500 open-ended behavioral samples rated by 151 humans and an LLM-judge ensemble. Humans and judges agree about model behavior ($\bar{r} = .51$), but self-report barely tracks human ratings ($\bar{r} = .09$, 95% CI $[-.07, .18]$) or rater-free text measures, and correcting for criterion unreliability leaves four of five factors near zero. Verbosity is the partial exception ($r = .40$, 71% of its reliability ceiling). On Responsiveness, self-report tracks LLM judges more than humans ($r = .53$ vs. $.18$; Steiger $p = .04$), and controlling for length and formatting does not remove this: agreement between LLM judges and LLM self-report is weak evidence that either tracks human judgment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
