量化 LLM 模拟中的人格效应
Quantifying the Persona Effect in LLM Simulations
AI总结：
本文量化人格变量对 LLM 模拟人类主观标注的影响，发现其解释方差不足10%，但人格提示可带来统计显著改进，并与人格—标注相关性呈线性关系。
摘要：
大型语言模型（LLM）在模拟人类语言和行为方面展现出显著前景。本研究考察了整合人格变量（人口统计、社会和行为因素）如何影响 LLM 模拟多元视角的能力。我们发现，在现有主观 NLP 数据集中，人格变量在标注中的方差占比不足 10%。尽管如此，通过提示将人格变量纳入 LLM，仍可带来适度但具有统计显著性的改进。人格提示在许多标注者存在分歧、但其分歧相对较小的样本中最为有效。值得注意的是，在我们的设定中发现了一种线性关系：人格变量与人类标注之间的相关性越强，使用人格提示的 LLM 预测就越准确。在零样本设定下，一个强大的 70b 模型借助人格提示，可捕捉基于真实标注训练的线性回归所能达到标注方差的 81%。然而，对于大多数人格变量解释力有限的主观 NLP 数据集，人格提示的收益也有限。
英文摘要：
Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. This study investigates how integrating persona variables-demographic, social, and behavioral factors-impacts LLMs' ability to simulate diverse perspectives. We find that persona variables account for <10% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating persona variables via prompting in LLMs provides modest but statistically significant improvements. Persona prompting is most effective in samples where many annotators disagree, but their disagreements are relatively minor. Notably, we find a linear relationship in our setting: the stronger the correlation between persona variables and human annotations, the more accurate the LLM predictions are using persona prompting. In a zero-shot setting, a powerful 70b model with persona prompting captures 81% of the annotation variance achievable by linear regression trained on ground truth annotations. However, for most subjective NLP datasets, where persona variables have limited explanatory power, the benefits of persona prompting are limited.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
