arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.00461cs.CYcs.CL
收藏

谁的人设?LLM研究中的人设实验及透明化路径

Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

Jan Batzner, Volker Stocker, Bingjun Tang, Anusha Natarajan, Qinhao Chen, Stefan Schmid, Gjergji Kasneci

AI总结:

本文探讨了LLM研究中合成人设实验的代表性与生态效度问题,提出透明化检查表以提升评估的严谨性和实证性。

摘要:

合成人设实验已成为大语言模型对齐研究中一种突出的方法,但这些人设的代表性和生态效度在不同研究之间差异显著。通过回顾2023年至2025年间发表于顶级自然语言处理和人工智能会议上的63篇同行评审论文,我们揭示了一个关键缺口:在基于人设的实验中,任务和目标人群往往未明确界定,尽管个性化本质上依赖于这些标准。我们的分析显示用户代表性的差异很大,大多数研究只关注有限的社会人口属性,只有35%的研究讨论了其LLM人设的代表性。基于我们的发现,我们引入了一张人设透明性检查表,强调代表性抽样、明确基于实证数据的 grounding,以及增强的生态效度。我们的工作既提供了当前实践的全面评估,也提供了改进语言模型对齐研究中基于人设评估严谨性和生态效度的实用指南。

英文摘要:

Synthetic personae experiments have become a prominent method in Large Language Model alignment research, yet the representativeness and ecological validity of these personae vary considerably between studies. Through a review of 63 peer-reviewed studies published between 2023 and 2025 in leading NLP and AI venues, we reveal a critical gap: task and population of interest are often underspecified in persona-based experiments, despite personalization being fundamentally dependent on these criteria. Our analysis shows substantial differences in user representation, with most studies focusing on limited sociodemographic attributes and only 35% discussing the representativeness of their LLM personae. Based on our findings, we introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity. Our work provides both a comprehensive assessment of current practices and practical guidelines to improve the rigor and ecological validity of persona-based evaluations in language model alignment research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑