面对分布变化的学习神经模型用于自然语言处理
Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
摘要：
训练强神经预测器以在特定数据集上执行一项任务的主导NLP范式已经在各种应用中实现了最先进的性能（例如，情感分类、基于跨度预测的问答或机器翻译）。然而，它建立在数据分布是平稳的假设之上，即。数据在训练和测试时都是从固定分布中采样的。这种训练方式与我们作为人类如何能够在不断变化的信息流中学习和操作不一致。此外，它不适合真实世界的用例，其中数据分布预计会在模型的生命周期中发生变化。本文的第一个目标是描述这种转变在自然语言处理背景下可能采取的不同形式，并提出基准和评估指标来衡量其对当前深度学习架构的影响。然后，我们继续采取措施来减轻分布转移对NLP模型的影响。为此，我们开发的方法的基础上的分布鲁棒优化框架的参数重构。从经验上讲，我们证明，这些方法产生更强大的模型上表现出的选择现实的问题。在本论文的第三部分，也是最后一部分，我们将探索如何有效地使现有模型适应新的领域或任务。我们的贡献，这一主题的灵感来自信息几何推导出一个新的梯度更新规则，减轻适应过程中的灾难性遗忘问题。
英文摘要：
The dominating NLP paradigm of training a strong neural predictor to perform one task on a specific dataset has led to state-of-the-art performance in a variety of applications (eg. sentiment classification, span-prediction based question answering or machine translation). However, it builds upon the assumption that the data distribution is stationary, ie. that the data is sampled from a fixed distribution both at training and test time. This way of training is inconsistent with how we as humans are able to learn from and operate within a constantly changing stream of information. Moreover, it is ill-adapted to real-world use cases where the data distribution is expected to shift over the course of a model's lifetime. The first goal of this thesis is to characterize the different forms this shift can take in the context of natural language processing, and propose benchmarks and evaluation metrics to measure its effect on current deep learning architectures. We then proceed to take steps to mitigate the effect of distributional shift on NLP models. To this end, we develop methods based on parametric reformulations of the distributionally robust optimization framework. Empirically, we demonstrate that these approaches yield more robust models as demonstrated on a selection of realistic problems. In the third and final part of this thesis, we explore ways of efficiently adapting existing models to new domains or tasks. Our contribution to this topic takes inspiration from information geometry to derive a new gradient update rule which alleviate catastrophic forgetting issues during adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
