测量所关心的：使用情境判断测试对AI进行心理测量评估
Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
AI总结：
本文通过情境判断测试和多维项目反应理论评估AI行为一致性，发现基于角色的AI行为稳定且可预测，提出更可靠的评估方法。
摘要：
Persona conditioning是广泛用于引导大型语言模型（LLM）行为的手段，但其是否诱导稳定的行为结构还是表面变化尚不明确。我们提出一个框架，利用情境判断测试（SJTs）、多维项目反应理论（MIRT）和结构化的合成角色，通过响应作为潜在行为变量的观测来测量一致的行为倾向。在大规模SJT和角色数据集中，我们发现基于角色的行为在多次运行中保持稳定，潜在特质分数可预测外部基准（如TruthfulQA、EmoBench），且MIRT揭示了一致的潜在结构。我们通过人工标注、基准评估和内部一致性分析验证了这些结果。我们解释这些特质不是人类性格，而是跨情境表现的稳定行为倾向。我们的结果表明，基于场景的心理测量评估为评估LLM行为提供了一种更可靠的替代方法，我们还发布了数据集以支持进一步研究。
英文摘要：
Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation. We propose a framework to measure consistent behavioral tendencies using situational judgment tests (SJTs), multidimensional item response theory (MIRT), and structured synthetic personas, treating responses as observations of latent behavioral variables. Across large-scale SJT and persona datasets, we find that persona-conditioned behaviors are stable across runs, latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure. We validate these results through human annotation, benchmark evaluation, and internal consistency analyses. We interpret these traits not as human personality, but as stable behavioral tendencies expressed across contexts. Our results show that scenario-based psychometric evaluation provides a more reliable alternative to classical self-report approaches for assessing LLM behavior, and we release datasets to support further study.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
