FraQ：用于联邦低秩适配的高效坐标空间重压缩方法
FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
浏览论文内容
中文总结 AI 辅助
FraQ是一种联邦低秩适配的高效坐标空间重压缩方法，可在接近未压缩基线准确率的同时减少下行通信，降低服务器端重压缩开销。
中文摘要 AI 辅助
采用低秩适配（LoRA）的联邦微调技术可实现大型语言模型（LLM）的高效协作适配，且无需集中私有数据。然而，LoRA的双因子参数化会在客户端间产生聚合不匹配问题：直接对因子取平均无法恢复其诱导更新的平均值。若在完整权重空间中形成精确聚合后再进行重压缩，可避免该不匹配，但对生成的稠密矩阵进行分解计算成本高且内存占用大。我们提出FraQ，一种用于联邦LoRA的高效坐标空间重压缩方法。该方法从精确表示聚合结果的堆叠因子出发，将其分解为正交基和紧凑坐标矩阵，再从小型Gram矩阵中恢复奇异谱，选择满足规定能量阈值的最小秩，最后通过基将选定的坐标子空间映射回去以构建全局适配器。在文本分类和常识推理基准上的实验表明，FraQ的准确率接近未压缩基线，同时大幅减少了下行链路通信，且服务器端重压缩开销较低。
英文摘要
Federated fine-tuning with Low-Rank Adaptation (LoRA) enables efficient collaborative adaptation of Large Language Models (LLMs) without centralizing private data. However, LoRA's two-factor parameterization creates an aggregation mismatch across clients: naively averaging the factors does not recover the average of their induced updates. This mismatch can be avoided by forming the exact aggregate in the full weight space and then recompressing it, but decomposing the resulting dense matrix is computationally expensive and memory-intensive. We propose FraQ, an efficient coordinate-space recompression method for federated LoRA. Starting from stacked factors that exactly represent the aggregate, FraQ factorizes it into an orthonormal basis and a compact coordinate matrix. It then recovers the singular spectrum from a small Gram matrix, selects the smallest rank satisfying a prescribed energy threshold, and maps the selected coordinate subspace back through the basis to construct the global adapter. Experiments on text classification and commonsense reasoning benchmarks show that FraQ achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.
