首页/论文/2004.14366

arXiv 2004.143662020-04cs.CLcs.LG

Elastic weight consolidation for better bias inoculation

University of Cambridge（剑桥大学）

James Thorne, Andreas Vlachos

备注：: Accepted at EACL 2021. Was previously submitted to arxiv with the title "Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation"

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

The biases present in training datasets have been shown to affect models for sentence pair classification tasks such as natural language inference (NLI) and fact verification. While fine-tuning models on additional data has been used to mitigate them, a common issue is that of catastrophic forgetting of the original training dataset. In this paper, we show that elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases while being less susceptible to catastrophic forgetting. In our evaluation on fact verification and NLI stress tests, we show that fine-tuning with EWC dominates standard fine-tuning, yielding models with lower levels of forgetting on the original (biased) dataset for equivalent gains in accuracy on the fine-tuning (unbiased) dataset.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Sequential Domain Adaptation through Elastic Weight Consolidation for Sentiment Analysis

Samsung R&D Institute India - Bangalore（三星印度研发中心-班加罗尔）

Elastic Weight Consolidation (EWC) is a technique used in overcoming catastrophic forgetting between successive tasks trained on a neural network. We use this phenomenon of information sharing between tasks for domain adaptation. Training data for tasks such as sentiment analysis (SA) may not be fairly represented across multiple domains. Domain Adaptation (DA) aims to build algorithms that leverage information from source domains to facilitate performance on an unseen target domain. We propose a model-independent framework - Sequential Domain Adaptation (SDA). SDA draws on EWC for training on…

2020-07-19cs.CL

Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation

University of Cambridge（剑桥大学）

Despite strong performance in many sequence-to-sequence tasks, autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between the ground-truth prefixes used during training and the model-generated prefixes used at inference time. Scheduled sampling is a simple and empirically successful approach which addresses this issue by incorporating model-generated prefixes into training. However, it has been argued that it is an inconsistent training objective leading to models ignoring the prefixes altogether. In this paper, we conduct systemati…

2023-01-10cs.CLcs.LG

Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation

Charles University（查理大学）；Faculty of Mathematics and Physics（数学与物理学院）

This work presents our ongoing research of unsupervised pretraining in neural machine translation (NMT). In our method, we initialize the weights of the encoder and decoder with two language models that are trained with monolingual data and then fine-tune the model on parallel data using Elastic Weight Consolidation (EWC) to avoid forgetting of the original language modeling tasks. We compare the regularization by EWC with the previous work that focuses on regularization by language modeling objectives. The positive result is that using EWC with the decoder achieves BLEU scores similar to the …

2020-10-19cs.CL

Sequential Targeting: an incremental learning approach for data imbalance in text classification

Korea University（高丽大学）；NAVER；Yonsei University（延世大学）；Naver Corp.（Naver公司）；TU Kaiserslautern（凯泽斯劳滕工业大学）；Korea Institute of Science and Technology Europe Forschungsgesellschaft mbH（韩国科学技术研究院欧洲研究所有限公司）

Classification tasks require a balanced distribution of data to ensure the learner to be trained to generalize over all classes. In real-world datasets, however, the number of instances vary substantially among classes. This typically leads to a learner that promotes bias towards the majority group due to its dominating property. Therefore, methods to handle imbalanced datasets are crucial for alleviating distributional skews and fully utilizing the under-represented data, especially in text classification. While addressing the imbalance in text data, most methods utilize sampling methods on t…

2020-11-23cs.CLcs.IRcs.LG

An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training

Amazon（亚马逊）

Pre-training large language models has become a standard in the natural language processing community. Such models are pre-trained on generic data (e.g. BookCorpus and English Wikipedia) and often fine-tuned on tasks in the same domain. However, in order to achieve state-of-the-art performance on out of domain tasks such as clinical named entity recognition and relation extraction, additional in domain pre-training is required. In practice, staged multi-domain pre-training presents performance deterioration in the form of catastrophic forgetting (CF) when evaluated on a generic benchmark such …

2020-10-01cs.CL

迈向持续学习新语言

Towards continually learning new languages

Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Carnegie Mellon University（卡内基梅隆大学）

本研究结合权重因子分解与弹性权重巩固，实现从10种语言持续学习至26种语言，消除灾难性遗忘，性能接近一次性训练所有语言。

2024-07-17cs.CLcs.SDeess.AS

更多相关论文收起更多论文

针对翻译任务微调大语言模型：通用遗忘缓解方法无法保持机器翻译特有的指令遵循能力

Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Following

AppTek GmbH（AppTek公司）；RWTH Aachen University（亚琛工业大学）；Applied Science Private University（应用科学私立大学）

研究针对翻译微调中的灾难性遗忘，发现通用遗忘缓解方法（如弹性权重巩固）虽能保持通用能力，但无法保持机器翻译特有的指令遵循（如形式、性别、长度控制），仅数据混合方法有效但泛化性有限。

2026-09-24cs.CLcs.LG

学习且不遗忘：向ASR基础模型添加新语言

Learn and Don't Forget: Adding a New Language to ASR Foundation Models

ALTA Institute, Machine Intelligence Lab, Department of Engineering, Cambridge University, UK

研究向ASR基础模型添加新语言同时保持原语言性能的方法，比较了软语言代码调优、软提示调优、LoRA和EWC等策略，发现直接微调效果最好但有损原语言能力，EWC能针对特定语言平衡此问题，仅用适应参数可维持原能力但新语言性能受限。

2024-09-24cs.CLcs.LGcs.SD

语音识别模型终身训练的顺序编辑

Sequential Editing for Lifelong Training of Speech Recognition Models

AWS AI Labs（AWS人工智能实验室）

提出顺序模型编辑方法用于ASR系统的终身训练，无需访问先前数据或额外参数，在CommonVoice多口音数据集上相较微调基线实现高达15%的WERR，且优于其他LLL技术。

2024-09-18cs.CLcs.SDeess.AS

Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem

University of Cambridge（剑桥大学）

Training data for NLP tasks often exhibits gender bias in that fewer sentences refer to women than to men. In Neural Machine Translation (NMT) gender bias has been shown to reduce translation quality, particularly when the target language has grammatical gender. The recent WinoMT challenge set allows us to measure this effect directly (Stanovsky et al, 2019). Ideally we would reduce system bias by simply debiasing all data prior to training, but achieving this effectively is itself a challenge. Rather than attempt to create a `balanced' dataset, we use transfer learning on a small set of trust…

2020-07-09cs.CL

Domain Expansion in DNN-based Acoustic Models for Robust Speech Recognition

University of Texas at Dallas（德克萨斯大学达拉斯分校）

Training acoustic models with sequentially incoming data -- while both leveraging new data and avoiding the forgetting effect-- is an essential obstacle to achieving human intelligence level in speech recognition. An obvious approach to leverage data from a new domain (e.g., new accented speech) is to first generate a comprehensive dataset of all domains, by combining all available data, and then use this dataset to retrain the acoustic models. However, as the amount of training data grows, storing and retraining on such a large-scale dataset becomes practically impossible. To deal with this p…

2019-10-01cs.CLcs.LGeess.AS