arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.12494cs.CLcs.AI
收藏

GenderCARE:一个评估和减少大型语言模型中性别偏见的综合框架

GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models

University of Science and Technology of China(中国科学技术大学);A*STAR(新加坡科技研究局);Beihang University(北京航空航天大学);Nanyang Technological University(南洋理工大学)

Kunsheng Tang, Wenbo Zhou, Jie Zhang, Aishan Liu, Gelei Deng, Shuai Li, Peigui Qi, Weiming Zhang, Tianwei Zhang, Nenghai Yu

AI总结:

GenderCARE提出综合框架,含新标准、对基准GenderPair和去偏技术,显著减少LLMs性别偏见,平均超35%,最高超90%,且语言任务性能损失低于2%。

摘要:

大型语言模型(LLMs)在自然语言生成方面展现了卓越的能力,但也被观察到会放大社会偏见,尤其是与性别相关的偏见。针对这一问题,已有多个基准被提出用于评估LLMs中的性别偏见。然而,这些基准往往缺乏实际灵活性或无意中引入偏见。为了解决这些不足,我们引入了GenderCARE,一个综合框架,包含创新的标准、偏见评估、减少技术和评估指标,用于量化和缓解LLMs中的性别偏见。首先,我们为性别平等基准建立了开创性标准,涵盖包容性、多样性、可解释性、客观性、稳健性和现实性等维度。在这些标准的指导下,我们构建了GenderPair,一种新颖的基于对的基准,旨在全面评估LLMs中的性别偏见。我们的基准提供了标准化和现实的评估,包括先前被忽视的性别群体,如跨性别者和非二元性别者。此外,我们开发了有效的去偏技术,结合反事实数据增强和专门的微调策略,以减少LLMs中的性别偏见,同时不损害其整体性能。大量实验表明,在各种性别偏见基准上显著减少,减少幅度最高超过90%,在17个不同的LLMs上平均超过35%。重要的是,这些减少在主流水语言任务中带来的变异性极小,保持在2%以下。通过提供现实的评估和定制的性别偏见减少,我们希望我们的GenderCARE能代表在LLMs中实现公平和公正的重要一步。更多细节请访问https://github.com/kstanghere/GenderCARE-ccs24。

英文摘要:

Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but they have also been observed to magnify societal biases, particularly those related to gender. In response to this issue, several benchmarks have been proposed to assess gender bias in LLMs. However, these benchmarks often lack practical flexibility or inadvertently introduce biases. To address these shortcomings, we introduce GenderCARE, a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics for quantifying and mitigating gender bias in LLMs. To begin, we establish pioneering criteria for gender equality benchmarks, spanning dimensions such as inclusivity, diversity, explainability, objectivity, robustness, and realisticity. Guided by these criteria, we construct GenderPair, a novel pair-based benchmark designed to assess gender bias in LLMs comprehensively. Our benchmark provides standardized and realistic evaluations, including previously overlooked gender groups such as transgender and non-binary individuals. Furthermore, we develop effective debiasing techniques that incorporate counterfactual data augmentation and specialized fine-tuning strategies to reduce gender bias in LLMs without compromising their overall performance. Extensive experiments demonstrate a significant reduction in various gender bias benchmarks, with reductions peaking at over 90% and averaging above 35% across 17 different LLMs. Importantly, these reductions come with minimal variability in mainstream language tasks, remaining below 2%. By offering a realistic assessment and tailored reduction of gender biases, we hope that our GenderCARE can represent a significant step towards achieving fairness and equity in LLMs. More details are available at https://github.com/kstanghere/GenderCARE-ccs24.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑