arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.10652cs.CLcs.AI
收藏

基于自适应合成模块的连续序列生成

Continual Sequence Generation with Adaptive Compositional Modules

Georgia Institute of Technology(佐治亚理工学院);Google(谷歌)

Yanzhe Zhang, Xuezhi Wang, Diyi Yang

摘要:

持续学习对于需要快速调整模型以适应新任务而不忘记旧任务知识的实际部署至关重要。现有的连续序列生成工作要么总是重用现有的参数来学习新的任务,这是脆弱的灾难性遗忘对不同的任务,或盲目地添加新的参数,这可能会阻止类似的任务之间的知识共享。为了两全其美,在这项工作中,我们提出了连续的序列生成自适应组合模块自适应地添加模块在Transformer架构和组成新的任务的新旧模块。我们还将伪经验重放,以促进知识转移,在这些共享模块。在不同序列的生成任务上的实验结果表明,该框架可以根据任务相似性自适应地添加模块或重用模块,在性能和参数效率方面均优于最先进的基线。我们在https://github.com/GT-SALT/Adaptive-Compositional-Modules上公开我们的代码。

英文摘要:

Continual learning is essential for real-world deployment when there is a need to quickly adapt the model to new tasks without forgetting knowledge of old tasks. Existing work on continual sequence generation either always reuses existing parameters to learn new tasks, which is vulnerable to catastrophic forgetting on dissimilar tasks, or blindly adds new parameters for every new task, which could prevent knowledge sharing between similar tasks. To get the best of both worlds, in this work, we propose continual sequence generation with adaptive compositional modules to adaptively add modules in transformer architectures and compose both old and new modules for new tasks. We also incorporate pseudo experience replay to facilitate knowledge transfer in those shared modules. Experiment results on various sequences of generation tasks show that our framework can adaptively add modules or reuse modules based on task similarity, outperforming state-of-the-art baselines in terms of both performance and parameter efficiency. We make our code public at https://github.com/GT-SALT/Adaptive-Compositional-Modules.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑