论语言模型中真实性与政治偏见的关系
On the Relationship between Truth and Political Bias in Language Models
AI总结：
本研究分析语言模型中真实性与政治偏见的关系，发现基于主流真实性数据集训练的奖励模型及现有开源奖励模型存在左倾偏见且规模越大偏见越显著，引发相关重要问题探讨。
摘要：
语言模型对齐研究通常试图确保模型不仅有用且无害，还需真实且无偏见。然而，同时优化这些目标可能会掩盖改进某一方面对其他方面的影响。在本研究中，我们聚焦分析语言模型对齐与政治学中两个关键概念的关系：真实性和政治偏见。我们在多种主流真实性数据集上训练奖励模型，随后评估其政治偏见。研究发现，基于这些数据集优化奖励模型的真实性往往会导致左倾政治偏见；现有开源奖励模型（即基于标准人类偏好数据集训练的模型）已呈现类似偏见，且模型规模越大，偏见越显著。这些结果引发了关于用于表征真实性的数据集、模型同时实现真实与政治无偏对齐的潜在局限，以及语言模型对真实与政治关系捕捉的重要问题。
英文摘要：
Language model alignment research often attempts to ensure that models are not only helpful and harmless, but also truthful and unbiased. However, optimizing these objectives simultaneously can obscure how improving one aspect might impact the others. In this work, we focus on analyzing the relationship between two concepts essential in both language model alignment and political science: truthfulness and political bias. We train reward models on various popular truthfulness datasets and subsequently evaluate their political bias. Our findings reveal that optimizing reward models for truthfulness on these datasets tends to result in a left-leaning political bias. We also find that existing open-source reward models (i.e., those trained on standard human preference datasets) already show a similar bias and that the bias is larger for larger models. These results raise important questions about the datasets used to represent truthfulness, potential limitations of aligning models to be both truthful and politically unbiased, and what language models capture about the relationship between truth and politics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
