稀疏适配器融合用于自然语言处理中的持续学习
Sparse Adapter Fusion for Continual Learning in NLP
AI总结：
本研究提出稀疏适配器融合方法，通过动态融合旧新适配器提升自然语言处理中持续学习的效率与性能。
摘要：
自然语言处理中的持续学习在适应不断变化的数据和防止灾难性遗忘中起着关键作用。尽管取得了显著进展，现有方法仍面临挑战，例如在不同任务之间重用参数效率低下，当任务不相似时会引发灾难性遗忘，以及为每个任务引入不必要的新参数，这阻碍了相似任务之间的知识共享。为了解决这些问题，我们提出了一种稀疏适配器融合方法（SAFM），该方法动态融合旧和新适配器以解决这些挑战。SAFM分为两个阶段：决策阶段和调优阶段。在决策阶段，SAFM决定是否引入新适配器、重用现有适配器或添加空适配器。架构搜索过程设计为优先重用或添加空适配器，以最小化参数消耗并最大化重用。在调优阶段，SAFM特别促进层间损失以鼓励适配器之间的区分，有效捕捉同一任务内的知识。实验结果一致显示，SAFM优于最先进的（SOTA）方法，实现了与之相当的性能，同时仅使用不到60%的参数。
英文摘要：
Continual learning in natural language processing plays a crucial role in adapting to evolving data and preventing catastrophic forgetting. Despite significant progress, existing methods still face challenges, such as inefficient parameter reuse across tasks, risking catastrophic forgetting when tasks are dissimilar, and the unnecessary introduction of new parameters for each task, which hampers knowledge sharing among similar tasks. To tackle these issues, we propose a Sparse Adapter Fusion Method (SAFM), which dynamically fuses old and new adapters to address these challenges. SAFM operates in two stages: the decision stage and the tuning stage. In the decision stage, SAFM determines whether to incorporate a new adapter, reuse an existing one, or add an empty adapter. The architecture search procedure, designed to prioritize reusing or adding empty adapters, minimizes parameter consumption and maximizes reuse. In the tuning stage, SAFM especially facilitates a layer-wise loss to encourage differentiation between adapters, effectively capturing knowledge within the same task. Experimental results consistently show that SAFM outperforms state-of-the-art (SOTA) methods, achieving comparable performance while utilizing less than 60% of the parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
