CALM：用于语言建模的连续自适应学习
CALM: Continuous Adaptive Learning for Language Modeling
摘要：
训练大型语言表示模型已经成为自然语言处理社区的标准。这允许对任何数量的特定任务进行微调，然而，这些大的高容量模型可以继续对特定于域的未标记数据进行训练，以使初始化对于监督任务更加鲁棒。我们证明，在实践中，这些预先训练的模型表现出性能恶化的灾难性遗忘的形式时，从一个一般的域，如GLUE的任务进行评估。在这项工作中，我们提出了CALM，持续自适应学习语言建模：技术来渲染模型，保留跨多个领域的知识。通过这些方法，我们能够减少由任务特定模型引入的监督任务之间的性能差距，我们使用生物医学和临床领域中的持续学习设置来证明这一点。
英文摘要：
Training large language representation models has become a standard in the natural language processing community. This allows for fine tuning on any number of specific tasks, however, these large high capacity models can continue to train on domain specific unlabeled data to make initialization even more robust for supervised tasks. We demonstrate that in practice these pre-trained models present performance deterioration in the form of catastrophic forgetting when evaluated on tasks from a general domain such as GLUE. In this work we propose CALM, Continuous Adaptive Learning for Language Modeling: techniques to render models which retain knowledge across multiple domains. With these methods, we are able to reduce the performance gap across supervised tasks introduced by task specific models which we demonstrate using a continual learning setting in biomedical and clinical domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
