面向心理学中鲁棒大语言模型研究的有效性导向工作流程
A validity-guided workflow for robust large language model research in psychology
AI总结：
本文提出一个六阶段工作流程，通过整合心理测量与因果推断，提升心理学中大语言模型研究的有效性，强调研究目标、计算工具验证、实验设计、透明执行、数据分析和结果报告的全面性。
摘要：
大型语言模型（LLMs）正迅速被整合到心理学研究中作为研究工具、评估目标、人类模拟器和认知模型。然而，近期证据显示存在严重测量不可靠性：人格评估在因子分析下崩溃，道德偏好随标点变化反转，理论思维准确性在微小改写中变化显著。这些“测量幽灵”——伪装成心理现象的统计伪影——威胁着越来越多研究的有效性。基于整合心理测量与因果推断的双重有效性框架，本文提出一个六阶段工作流程，将有效性要求扩展到研究雄心——使用LLM进行文本编码需要基本可靠性和准确性，而关于心理属性的主张则需全面的构念验证。研究人员必须（1）明确研究目标和相应有效性要求，（2）通过心理测量测试开发和验证计算工具，（3）设计控制计算混淆因素的实验，（4）执行透明的协议，（5）使用适用于非独立观察的数据分析方法，（6）在已证明的边界内报告发现并利用结果完善理论。通过“LLM自我性”模型评估示例，展示了系统验证如何区分真正的计算现象和测量伪影。通过建立经过验证的计算工具和透明实践，该工作流程为AI心理学研究构建稳健的经验基础提供了路径。
英文摘要：
Large language models (LLMs) are rapidly being integrated into psychological and behavioral research as research tools, evaluation targets, human simulators, and cognitive models. Yet recent evidence reveals severe measurement unreliability: personality assessments degenerate under factor analysis, moral preferences reverse with punctuation changes, and theory-of-mind accuracy varies widely with trivial rephrasing. These "measurement phantoms"--statistical artifacts masquerading as psychological phenomena--threaten the validity of a growing body of research. Guided by the dual-validity framework that integrates psychometrics with causal inference, we present a six-stage workflow that scales validity requirements to research ambition--using LLMs to code text requires basic reliability and accuracy, whereas claims about psychological properties demand comprehensive construct validation. Researchers must (1) explicitly define their research goal and corresponding validity requirements, (2) develop and validate computational instruments through psychometric testing, (3) design experiments that control for computational confounds, (4) execute protocols transparently, (5) analyze data with methods appropriate for non-independent observations, and (6) report findings within boundaries and use results to refine theory. We illustrate the workflow through an example of model evaluation--"LLM selfhood"--showing how systematic validation can distinguish genuine computational phenomena from measurement artifacts. By establishing validated computational instruments and transparent practices, this workflow provides a path toward building a robust empirical foundation for AI psychology research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
