大规模人口统计学提示：当更多属性损害大语言模型与人类的一致性时
Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement
浏览论文内容
中文总结 AI 辅助
研究人口统计学属性作提示线索对LLM预测与人类注释一致性的影响，通过五个开源LLM在五个任务中实验，发现一致性与属性数量有关，受属性可学习性等因素影响，表明人口统计学提示效用依赖上下文。
中文摘要 AI 辅助
我们研究了作为提示线索提供的注释者人口统计学属性如何在五个任务中塑造大语言模型（LLM）预测与人类注释之间的一致性。使用五个开源LLM，我们系统地改变提示中人口统计学成分的数量和组成，涵盖从单属性到全属性配置的每种组合。我们的实验揭示了三个主要发现。首先，一致性在一到三个高信号属性时达到峰值，在全属性集下会下降，确立了一个明确的过度指定阈值。其次，人口统计学对人类注释的总体影响程度并不能预测哪些属性会改善LLM的一致性；相反，每个属性注释信号的可学习性和方向一致性都需要共同考虑。第三，神经元探测表明，只有在连贯的注释信号下，专门的激活才与一致性增益相关，而且仅激活量并不意味着可操纵性。这些结果共同表明，人口统计学提示不是一种单一的干预措施：其效用高度依赖于上下文，由属性信号质量、任务特征和模型架构塑造。
英文摘要
We investigate how annotator demographic attributes, supplied as prompt cues, shape the alignment between large language model (LLM) predictions and human annotations across five tasks. Using five open-source LLMs, we systematically vary the number and composition of demographic components in the prompt, spanning every combination from single-attribute through full-attribute configurations. Our experiments reveal three principal findings. First, alignment consistently peaks with one to three high-signal attributes and degrades under the full attribute set, establishing a clear over-specification threshold. Second, the overall magnitude of demographic influence on human annotations does not predict which attributes improve LLM alignment; instead, both the learnability and the directional coherence of each attribute's annotation signal need to be considered jointly. Third, neuron probing reveals that specialized activation correlates with alignment gains only under coherent annotation signals, and that activation volume alone does not imply steerability. Together, these results demonstrate that demographic prompting is not a monolithic intervention: its utility is highly context-dependent, shaped by attribute signal quality, task characteristics, and model architecture.
