对大型语言模型中社会期望反应进行量化与缓解:一种匹配可欲性的等级强制选择心理测量研究
Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study
AI总结:
本文提出了一种心理测量框架,用于量化和缓解LLM问卷评估中的社会期望反应偏差,通过匹配可欲性来减少偏差并保持目标特征的恢复。
摘要:
人类自述问卷越来越多地用于NLP中,以基准测试和审计大型语言模型(LLMs),从个性一致性到安全性和偏见评估。然而,这些工具假定诚实作答;在评估情境中,LLMs可能倾向于社会偏好答案——一种社会期望反应(SDR)——偏转问卷得出的分数和下游结论。我们提出了一种心理测量框架,用于量化和缓解LLM问卷评估中的SDR。为了量化SDR,相同的清单在HONEST与FAKE-GOOD指令下进行测试,SDR计算为从项目反应理论(IRT)估计的潜在分数中得到的方向校正标准化效应量。这使能够跨构念和回答格式进行比较,以及与人类指导欺骗基准进行比较。为了缓解,我们构建了一个大五清单,通过受约束优化从项目池中选择30个跨领域对以匹配可欲性。在九个遵循指令的LLM上,基于合成角色的评估显示,李克特式问卷显示出持续的SDR,而匹配可欲性的GFC显著削弱SDR,同时在恢复预期角色特征方面大体上保持不变。这些结果突显了模型依赖的SDR恢复权衡,并促使SDR意识的报告实践,以用于LLM的问卷基准测试和审计。
英文摘要:
Human self-report questionnaires are increasingly used in NLP to benchmark and audit large language models (LLMs), from persona consistency to safety and bias assessments. Yet these instruments presume honest responding; in evaluative contexts, LLMs can instead gravitate toward socially preferred answers-a form of socially desirable responding (SDR)-biasing questionnaire-derived scores and downstream conclusions. We propose a psychometric framework to quantify and mitigate SDR in questionnaire-based evaluation of LLMs. To quantify SDR, the same inventory is administered under HONEST versus FAKE-GOOD instructions, and SDR is computed as a direction-corrected standardized effect size from item response theory (IRT)-estimated latent scores. This enables comparisons across constructs and response formats, as well as against human instructed-faking benchmarks. For mitigation, we construct a graded forced-choice (GFC) Big Five inventory by selecting 30 cross-domain pairs from an item pool via constrained optimization to match desirability. Across nine instruction-following LLMs evaluated on synthetic personas with known target profiles, Likert-style questionnaires show consistently large SDR, whereas desirability-matched GFC substantially attenuates SDR while largely preserving the recovery of the intended persona profiles. These results highlight a model-dependent SDR-recovery trade-off and motivate SDR-aware reporting practices for questionnaire-based benchmarking and auditing of LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。