根塑造果实：论对齐语言模型中性别排他性伤害的持续存在
The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
AI总结：
本文研究对齐是否延续基础模型的性别多元偏见，评估16个DPO阶段模型，发现SFT会放大污名化和不肯定性别语言，并提出隐式奖励偏见测量框架。
摘要：
自然语言助手旨在为用户提供有帮助的回答，同时避免有害输出，这主要通过对人类偏好的对齐来实现。然而，人们对于对齐技术是否可能无意中延续甚至放大从其对齐前基础模型继承的有害偏见理解有限。流行的偏好微调模型中偏见评估基准的选择使这一问题更加复杂，这些基准主要关注二元性别等主导社会类别，从而限制了对影响代表性不足群体的偏见的认识。为解决这一空白，我们以跨性别者、非二元性别者及其他性别多元身份为中心，研究对齐过程如何与LLM中预先存在的性别多元偏见相互作用。我们的主要贡献包括：1）对主流偏好微调LLM的偏见评估方式进行全面综述，强调性别多元代表性方面的关键缺口；2）系统评估跨越Direct Preference Optimization（DPO）阶段的16个模型中的性别多元偏见，揭示流行偏见基准未能检测到的伤害；3）提出一个灵活框架，用于测量隐式奖励信号中的有害偏见，该框架可应用于其他社会情境。研究结果表明，DPO对齐模型对supervised finetuning（SFT）特别敏感，并可能放大其基础模型中两种现实世界的性别多元伤害：污名化和不肯定性别的语言。我们最后提出针对DPO及更广泛对齐实践的建议，倡导采用社区知情的偏见评估框架，以更有效地识别和解决LLM中代表性不足的伤害。
英文摘要：
Natural-language assistants are designed to provide users with helpful responses while avoiding harmful outputs, largely achieved through alignment to human preferences. Yet there is limited understanding of whether alignment techniques may inadvertently perpetuate or even amplify harmful biases inherited from their pre-aligned base models. This issue is compounded by the choice of bias evaluation benchmarks in popular preference-finetuned models, which predominantly focus on dominant social categories, such as binary gender, thereby limiting insights into biases affecting underrepresented groups. Towards addressing this gap, we center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias in LLMs. Our key contributions include: 1) a comprehensive survey of bias evaluation modalities across leading preference-finetuned LLMs, highlighting critical gaps in gender-diverse representation, 2) systematic evaluation of gender-diverse biases across 16 models spanning Direct Preference Optimization (DPO) stages, uncovering harms popular bias benchmarks fail to detect, and 3) a flexible framework for measuring harmful biases in implicit reward signals applicable to other social contexts. Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning (SFT), and can amplify two forms of real-world gender-diverse harms from their base models: stigmatization and gender non-affirmative language. We conclude with recommendations tailored to DPO and broader alignment practices, advocating for the adoption of community-informed bias evaluation frameworks to more effectively identify and address underrepresented harms in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
