任务无关的低秩残差适应用于高效的联邦持续微调
Task-agnostic Low-rank Residual Adaptation for Efficient Federated Continual Fine-Tuning
AI总结：
本文提出Fed-TaLoRA，通过任务无关适应、后聚合模型校准和策略性低秩适应放置，解决联邦持续微调中的遗忘、参数增长和聚合不一致问题，提升效率。
摘要：
联邦参数高效微调（Fed-PEFT）通过仅更新少量参数实现大预训练模型的轻量级适应。然而，Fed-PEFT方法通常假设固定标签空间和静态下游任务，这在现实应用中受限，因为客户端随时间不断遇到新类别。这导致了联邦持续微调（FCFT）问题的出现。在FCFT中，客户端协作微调预训练模型，每个客户端随时间观察到不同的新类别，且推理时无法获取任务身份。FCFT具有挑战性，因为它同时面临非IID客户端数据分布下的严重遗忘、参数增长和任务特定推理，以及异质客户端间的聚合不一致。为此，我们提出联邦任务无关低秩残差适应（Fed-TaLoRA），一种基于任务无关适应、后聚合模型校准和策略性低秩适应放置的新型方法。Fed-TaLoRA在连续任务中持续微调单一共享模块以避免任务特定参数增长，并进一步引入理论基础的残差权重更新机制以校准聚合的全局模型并提高聚合保真度。我们提供了Fed-TaLoRA收敛性和聚合行为的理论分析。在四个基准数据集上的广泛实验表明，Fed-TaLoRA在减少通信和计算成本的同时，始终优于强基线。
英文摘要：
Federated Parameter-Efficient Fine-Tuning (Fed-PEFT) enables lightweight adaptation of large pre-trained models in federated learning settings by updating only a small subset of parameters. However, Fed-PEFT methods typically assume a fixed label space and static downstream tasks, which is restrictive in realistic application scenarios where clients continuously encounter new classes over time. This leads to an emerging problem, known as \emph{Federated Continual Fine-Tuning} (FCFT). In FCFT, clients collaboratively fine-tune a pre-trained model over a sequence of tasks, where each client observes disjoint sets of new classes over time, and task identity is unavailable at inference time. FCFT is challenging because it simultaneously suffers from severe forgetting under non-IID client data distributions, parameter growth and task-specific inference caused by task-wise modules, and aggregation inconsistency across heterogeneous clients. To address these challenges, we propose Federated Task-agnostic Low-rank Residual Adaptation (Fed-TaLoRA), a novel approach for efficient FCFT built on task-agnostic adaptation, post-aggregation model calibration, and strategic low-rank adaptation placement. Fed-TaLoRA continuously fine-tunes a single shared module across sequential tasks to avoid task-wise parameter growth, and further introduces a theoretically grounded residual weight update mechanism to calibrate the aggregated global model and improve aggregation fidelity. We provide a theoretical analysis of the convergence and aggregation behavior of Fed-TaLoRA. Extensive experiments on four benchmark datasets demonstrate that Fed-TaLoRA consistently outperforms strong baselines while reducing communication and computation costs significantly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
