arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.08528cs.CL
收藏

小语言模型高效持续学习中的离散键值瓶颈

Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneck

Ulm University(乌尔姆大学);University of Southern Denmark(南部丹麦大学)

Andor Diera, Lukas Galke, Fabian Karl, Ansgar Scherp

AI总结:

本研究提出了一种适用于小语言模型的离散键值瓶颈结构,通过局部更新实现高效的持续学习,有效缓解灾难性遗忘并降低计算成本。

摘要:

持续学习在各种自然语言处理(NLP)任务中仍是一个挑战,因为使用新训练数据更新的模型往往有风险遗忘之前获得的知识。我们为仅编码器的语言模型引入了一个离散键值瓶颈(DKVB),通过局部更新实现高效的持续学习。受视觉中的离散键值瓶颈启发,我们考虑了新的和NLP特定的挑战。我们比较了不同的瓶颈架构用于NLP,并引入了一种新的、任务无关的离散键初始化技术。我们评估了我们的DKVB在四个持续学习场景中的NLP表现,并证明它减轻了灾难性遗忘。我们的实验表明,所提出的方法在性能上与流行的持续学习方法竞争,同时计算成本更低。此外,我们还显示DKVB在具有挑战性的单头持续学习场景中仍然有效,即使没有任务ID。

英文摘要:

Continual learning remains a challenge across various natural language processing (NLP) tasks, as models updated with new training data often risk catastrophic forgetting of previously acquired knowledge. We introduce a discrete key-value bottleneck (DKVB) for encoder-only language models, enabling efficient continual learning through localized updates. Inspired by a discrete key-value bottleneck in vision, we consider new and NLP-specific challenges. We compare different bottleneck architectures for NLP and introduce a new, task-independent initialization technique for the discrete keys. We evaluate our DKVB for NLP in four continual learning scenarios and show that it alleviates catastrophic forgetting. Our experiments demonstrate that the proposed approach achieves competitive performance compared to popular continual learning methods while incurring lower computational costs. Furthermore, we show that DKVB remains effective even in challenging single-head continual learning scenarios where no task ID is provided.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑