ELLA：为大语言模型适配器实现高效的终身学习
ELLA: Efficient Lifelong Learning for Adapters in Large Language Models
AI总结：
ELLA通过选择性子空间去相关原理，实现高效终身学习，无需数据重放或架构扩展，显著提升连续学习性能和零样本泛化能力。
摘要：
大语言模型（LLMs）在连续学习（CL）设置中依次适应新任务时会遭受严重的灾难性遗忘。现有方法从根本上存在局限：基于重放的方法不切实际且侵犯隐私，而严格正交的方法在规模扩大时会崩溃：每个新任务被投影到正交补集上，逐步减少残差自由度，通过禁止共享表示的重叠来消除前向转移。在本工作中，我们引入ELLA，一种基于选择性子空间去相关原理的训练框架。与禁止所有重叠不同，ELLA明确刻画了过去更新的结构，并惩罚沿着其高能、任务特定方向的对齐，同时在低能残差子空间中保留自由度以实现迁移。形式上，这通过在单个聚合更新矩阵上的轻量级正则化器实现。我们证明这种机制对应于一个各向异性收缩算子，该算子限制干扰，产生一个无论任务序列长度如何都保持内存和计算常数的惩罚。ELLA不需要数据重放、不需要架构扩展，且存储开销极小。实证上，它在三个流行的基准测试中实现了最先进的CL性能，相对准确率提升高达9.6%，且内存足迹小35倍。进一步，ELLA在不同架构上具有鲁棒的扩展性，并且积极增强模型在未见任务上的零样本泛化性能，建立了一个系统且可扩展的构造性终身LLM适应解决方案。
英文摘要：
Large Language Models (LLMs) suffer severe catastrophic forgetting when adapted sequentially to new tasks in a continual learning (CL) setting. Existing approaches are fundamentally limited: replay-based methods are impractical and privacy-violating, while strict orthogonality-based methods collapse under scale: each new task is projected onto an orthogonal complement, progressively reducing the residual degrees of freedom and eliminating forward transfer by forbidding overlap in shared representations. In this work, we introduce ELLA, a training framework built on the principle of selective subspace de-correlation. Rather than forbidding all overlap, ELLA explicitly characterizes the structure of past updates and penalizes alignments along their high-energy, task-specific directions, while preserving freedom in the low-energy residual subspaces to enable transfer. Formally, this is realized via a lightweight regularizer on a single aggregated update matrix. We prove this mechanism corresponds to an anisotropic shrinkage operator that bounds interference, yielding a penalty that is both memory- and compute-constant regardless of task sequence length. ELLA requires no data replay, no architectural expansion, and negligible storage. Empirically, it achieves state-of-the-art CL performance on three popular benchmarks, with relative accuracy gains of up to $9.6\%$ and a $35\times$ smaller memory footprint. Further, ELLA scales robustly across architectures and actively enhances the model's zero-shot generalization performance on unseen tasks, establishing a principled and scalable solution for constructive lifelong LLM adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
