arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.05489cs.CLcs.AI
收藏

基于正则化的持续学习文本分类

Continual Learning for Text Classification with Information Disentanglement Based Regularization

Georgia Institute of Technology(佐治亚理工学院);Zhejiang University(浙江大学);Google(谷歌)

Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, Diyi Yang

摘要:

持续学习变得越来越重要,因为它使NLP模型能够随着时间的推移不断学习和获得知识。以前的持续学习方法主要是为了保存以前任务中的知识,而没有太多地强调如何将模型很好地推广到新任务。在这项工作中,我们提出了一个基于信息去纠缠的正则化方法的持续学习的文本分类。我们提出的方法首先将文本隐藏空间分解为对所有任务通用的表示和对每个任务特定的表示,并进一步以不同的方式正则化这些表示,以更好地约束泛化所需的知识。我们还介绍了两个简单的辅助任务:下一句预测和任务ID预测,以学习更好的通用和特定的表示空间。在大规模基准测试上进行的实验表明,我们的方法在最先进的基线上具有各种序列和长度的连续文本分类任务中是有效的。我们已在https://github.com/GT-SALT/IDBR上公开发布了我们的代码。

英文摘要:

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑