RobustDebias: 使用分布鲁棒优化去偏语言模型
RobustDebias: Debiasing Language Models using Distributionally Robust Optimization
AI总结:
RobustDebias通过分布鲁棒优化在微调过程中减少语言模型的偏见,有效缓解偏见的同时保持性能
摘要:
预训练语言模型已被证明会表现出偏见和社会刻板印象。先前的工作主要集中在预训练过程中修改嵌入空间,这在大规模模型上并不可行。在任务特定数据集上微调预训练模型会降低模型性能并放大微调数据中的偏见。我们关注微调过程中的偏见放大问题,而不是昂贵的预训练过程,专注于BERT模型,因其在语言理解任务中被广泛使用。虽然经验风险最小化有效优化下游性能,但在微调过程中往往放大社会偏见。为此,我们提出了RobustDebias,一种新的机制,将分布鲁棒优化(DRO)适应到微调过程中去偏语言模型。我们的方法在多个 demographics 上微调 MLM 并推广到任何数据集或任务。在各种语言模型上的广泛实验显示,有显著的偏见缓解,且对性能影响很小。
英文摘要:
Pretrained language models have been shown to exhibit biases and social stereotypes. Prior work on debiasing these models has largely focused on modifying embedding spaces during pretraining, which is not scalable for large models. Fine-tuning pretrained models on task-specific datasets can both degrade model performance and amplify biases present in the fine-tuning data. We address bias amplification during fine-tuning rather than costly pretraining, focusing on BERT models due to their widespread use in language understanding tasks. While Empirical Risk Minimization effectively optimizes downstream performance, it often amplifies social biases during fine-tuning. To counter this, we propose \textit{RobustDebias}, a novel mechanism which adapts Distributionally Robust Optimization (DRO) to debias language models during fine-tuning. Our approach debiases models across multiple demographics during MLM fine-tuning and generalizes to any dataset or task. Extensive experiments on various language models show significant bias mitigation with minimal performance impact.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。