arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.03329cs.CLcs.AI
收藏

LAMOL:终身语言学习的语言建模

LAMOL: LAnguage MOdeling for Lifelong Language Learning

MIT(麻省理工学院);National Taiwan University(台湾大学)

Fan-Keng Sun, Cheng-Hao Ho, Hung-Yi Lee

摘要:

大多数关于终身学习的研究都适用于图像或游戏,而不是语言。我们提出了LAMOL,一个简单而有效的终身语言学习(LLL)的语言建模的基础上的方法。LAMOL重放以前任务的伪样本,而不需要额外的内存或模型容量。具体来说,LAMOL是一种语言模型,它同时学习解决任务并生成训练样本。当模型针对新任务进行训练时,它会生成先前任务的伪样本,以便与新任务的数据一起进行训练。结果表明,LAMOL可以防止灾难性遗忘,而没有任何不妥协的迹象,并可以执行五个非常不同的语言任务顺序只有一个模型。总的来说,LAMOL比以前的方法有相当大的优势,只比多任务处理差2-3%,这通常被认为是LLL的上限。源代码可在https://github.com/jojotenya/LAMOL上获得。

英文摘要:

Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/jojotenya/LAMOL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑