共享LoRA子空间用于近严格持续学习
Shared LoRA Subspaces for almost Strict Continual Learning
AI总结：
Share通过共享低秩子空间实现高效持续学习，减少参数和内存消耗，提升多任务适应能力。
摘要：
在现实世界部署中，高效且持续地适应大规模预训练模型以适应新任务至关重要，但受到灾难性遗忘和重新训练成本高的挑战。尽管参数高效调优方法如低秩适应（LoRA）降低了计算需求，但缺乏严格持续学习和知识整合的机制，不依赖数据回放或多个适配器。我们提出Share，一种新的参数高效持续微调方法，学习并动态更新一个单一的共享低秩子空间，使模型能够无缝适应多个任务和模态。Share构建了一个基础子空间，从过去任务中提取核心知识，并通过识别关键子空间方向逐步整合新信息。每个新任务的知识被整合到这个不断演变的子空间中，促进前向知识转移，同时最小化灾难性干扰。该方法在传统LoRA方法上实现了高达100倍的参数减少和281倍的内存节省，性能与联合训练模型相当。一个Share模型可以替代数百个任务特定的LoRA适配器，支持可扩展的异步持续学习。在图像分类、自然语言理解、3D姿态估计和文本到图像生成等实验中验证了其有效性，使Share成为大规模AI系统中终身学习的实用且可扩展的解决方案。
英文摘要：
Adapting large pretrained models to new tasks efficiently and continually is crucial for real-world deployment but remains challenging due to catastrophic forgetting and the high cost of retraining. While parameter-efficient tuning methods like low rank adaptation (LoRA) reduce computational demands, they lack mechanisms for strict continual learning and knowledge integration, without relying on data replay, or multiple adapters. We propose Share, a novel approach to parameter efficient continual finetuning that learns and dynamically updates a single, shared low-rank subspace, enabling seamless adaptation across multiple tasks and modalities. Share constructs a foundational subspace that extracts core knowledge from past tasks and incrementally integrates new information by identifying essential subspace directions. Knowledge from each new task is incorporated into this evolving subspace, facilitating forward knowledge transfer, while minimizing catastrophic interference. This approach achieves up to 100x parameter reduction and 281x memory savings over traditional LoRA methods, maintaining performance comparable to jointly trained models. A single Share model can replace hundreds of task-specific LoRA adapters, supporting scalable, asynchronous continual learning. Experiments across image classification, natural language understanding, 3D pose estimation, and text-to-image generation validate its effectiveness, making Share a practical and scalable solution for lifelong learning in large-scale AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
