Synthetic personae experiments have become a prominent method in Large Language Model alignment research, yet the representativeness and ecological validity of these personae vary considerably between studies. Through a review of 63 peer-reviewed studies published between 2023 and 2025 in leading NLP and AI venues, we reveal a critical gap: task and population of interest are often underspecified in persona-based experiments, despite personalization being fundamentally dependent on these criteria. Our analysis shows substantial differences in user representation, with most studies focusing on limited sociodemographic attributes and only 35% discussing the representativeness of their LLM personae. Based on our findings, we introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity. Our work provides both a comprehensive assessment of current practices and practical guidelines to improve the rigor and ecological validity of persona-based evaluations in language model alignment research.
Analysing Personal Attacks in U.S. Presidential Debates
School of Computer Science and Engineering, University of New South Wales(计算机科学与工程学院,新南威尔士大学);School of Mathematics and Statistics, University of New South Wales(数学与统计学学院,新南威尔士大学)
Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalization
Institute for Logic, Language and Computation, University of Amsterdam(逻辑、语言与计算研究所,阿姆斯特丹大学);Center for Language and Cognition, University of Groningen(语言与认知中心,格罗宁根大学)
From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models
Institute for Clarity in Documentation(文献清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康库尔中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室);The Thørväld Group(索尔沃尔德集团);The Kumquat Consortium(金桔联盟)
Exploring Big Five Personality and AI Capability Effects in LLM-Simulated Negotiation Dialogues
Aptima, Inc.(Aptima公司);Human Systems Engineering, Arizona State University(亚利桑那州立大学人类系统工程系);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)