作为人口统计学的大语言模型：社会人口提示法帮助谁，又伤害谁
LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurts
- Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）
- Almawave Labs（Almawave实验室）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究探究社会人口提示法对LLM主观任务评判的影响，发现其存在不对称性，会偏向多数群体、伤害少数群体，交叉特征会放大伤害，指令调优或为不对称性的来源。
AI中文摘要：
大语言模型（LLM）越来越多地被用作主观任务的评判者，在这类任务中，标注者之间存在分歧，相关问题不仅在于评判者的准确性，还在于其复刻谁的判断。社会人口提示法通过让评判者基于标注者的人口统计特征，使其判断与对应群体保持一致。我们测试这种一致性是否会在分布层面显现，在三种条件下，对比23个开放权重LLM在三项主观任务上的预测标签分布与真实标注者群体的分布：无人口统计信息、单一属性特征、以及涵盖性别、年龄、种族和教育的交叉性特征。研究得出三项发现：第一，未添加人口统计信息提示的评判者并非视角中立，模型最能复刻白人、受过大学教育的标注者的判断；第二，人口统计条件作用具有不对称性，它会将评判者推向多数群体，远离少数群体，在冒犯性任务上表现最为明显，交叉性特征会放大这种伤害；第三，通过对比基础模型与指令调优模型，我们发现指令调优可能是导致这种不对称性的原因。因此，使用人口统计条件作用来估计群体判断时需谨慎：条件作用会使预测偏离该方法常被用来服务的少数群体的参考分布。
英文摘要：
Large language models (LLMs) are increasingly used as judges for subjective tasks, where annotators disagree and the relevant question is not only how accurate a judge is, but whose judgments it reproduces. Sociodemographic prompting conditions the judge on an annotator's demographic profile to align its judgments with the corresponding group's. We test whether this alignment emerges distributionally, comparing the predicted label distributions of 23 open-weight LLMs on three subjective tasks against those of real annotator groups, under three conditions: no demographic information, single-attribute profiles, and intersectional profiles over gender, age, race, and education. Three findings emerge. First, a judge prompted with no demographics is not perspective-neutral: models best reproduce the judgments of White, college-educated annotators. Second, demographic conditioning is asymmetric: it moves the judge toward majority groups and away from minority groups, most strongly on offensiveness, where intersectional profiles amplify the harm. Third, by comparing base and instruct models we identify instruction-tuning as a possible source of the asymmetry. Demographic conditioning should therefore be used with caution to estimate group judgments: conditioning moves predictions away from the reference distributions of the minority groups the method is often invoked to serve.
