在不发生灾难性遗忘的情况下提升预训练语言模型的性别公平性
Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting
AI总结：
针对预训练语言模型性别偏见缓解中的灾难性遗忘问题，提出冻结预训练模型并学习性别相关提示的新方法GEEP，在提升性别公平性的同时大幅保持了下游任务性能。
摘要：
现有解决预训练语言模型性别偏见的研究，通常构建一个小型性别中立数据集，并使用此类数据对模型进行第二阶段预训练。然而，鉴于性别中立数据规模有限且关注点集中，在第二阶段预训练期间会发生灾难性遗忘。遗忘原始训练数据中的信息可能大幅损害模型的下游性能。在本研究中，我们通过在 GLUE 中使用通用 NLP 任务评估这些方法，实证表明灾难性遗忘确实存在于此类方法中。随后，我们提出一种新方法 GEnder Equality Prompt (GEEP)，以在减少遗忘的情况下提升预训练模型的性别公平性。GEEP 冻结预训练模型，并使用性别中立数据学习性别相关提示。实证结果表明，GEEP 不仅在性别公平性任务上取得了 SOTA 表现，而且遗忘更少，并在 GLUE 上以大幅优势取得了更好的表现。
英文摘要：
Existing studies addressing gender bias of pre-trained language models, usually build a small gender-neutral data set and conduct a second phase pre-training on the model with such data. However, given the limited size and concentrated focus of the gender-neutral data, catastrophic forgetting would occur during second-phase pre-training. Forgetting information in the original training data may damage the model's downstream performance by a large margin. In this work, we empirically show that catastrophic forgetting occurs in such methods by evaluating them with general NLP tasks in GLUE. Then, we propose a new method, GEnder Equality Prompt (GEEP), to improve gender fairness of pre-trained models with less forgetting. GEEP freezes the pre-trained model and learns gender-related prompts with gender-neutral data. Empirical results show that GEEP not only achieves SOTA performances on gender fairness tasks, but also forgets less and performs better on GLUE by a large margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
