发表机构
School of IT, Deakin University; Columbia University（迪肯大学信息技术学院; 哥伦比亚大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对联邦参数高效微调在非IID客户端异质性下的脆弱性，提出TRISHUL频谱控制框架，通过共享冻结多头低秩基、核范数近端收缩及非均匀分配适应头，提升了收敛性、稳定性和最终性能。
AI 中文摘要
联邦参数高效微调（PEFT）能在分散的边缘数据上高效通信地适配大型预训练模型，但在非独立同分布客户端异质性下仍很脆弱。在低秩适应（LoRA）中，不同客户端可能学习到局部有用但频谱未对齐的更新子空间，导致高方差聚合和较差的全局迁移。我们提出了TRISHUL，一个用于稳健联邦PEFT的频谱控制框架。TRISHUL遵循联邦学习无原始数据共享设置，但本身不提供正式隐私保证。它使用共享的冻结多头低秩基来获得紧凑核心更新的代数精确聚合，在上传前应用核范数近端收缩来抑制客户端特定的高秩频谱分量，并使用从预训练层容量导出的凹水填充预算规则在各层非均匀分配适应头。因为收缩仅在小核心矩阵上执行，TRISHUL在底层多头PEFT协议上增加的计算可忽略不计，且无额外的每轮通信。在包括CIFAR-100、SVHN、20 Newsgroups、MRQA和使用LLaMA3.2 - 1B的GLUE等视觉和语言基准测试中，TRISHUL相对于联邦LoRA基线提高了收敛性、稳定性和最终性能，在更强的异质性下有更大提升。
英文摘要
Federated parameter-efficient fine-tuning (PEFT) enables communication-efficient adaptation of large pretrained models on decentralized edge data, but it remains fragile under non-IID client heterogeneity. In low-rank adaptation (LoRA), different clients may learn locally useful but spectrally misaligned update subspaces, causing high-variance aggregation and poor global transfer. We propose TRISHUL, a spectral-control framework for robust federated PEFT. TRISHUL follows the FL no-raw-data-sharing setting but does not itself provide formal privacy guarantees. TRISHUL uses shared frozen multi-head low-rank bases to obtain algebraically exact aggregation of compact core updates, applies nuclear norm proximal shrinkage to suppress client-specific high-rank spectral components before upload, and allocates adaptation heads non-uniformly across layers using a concave water filling budget rule derived from pretrained layer capacity. Because shrinkage is performed only on small core matrices, TRISHUL adds negligible computation and no extra per-round communication over the underlying multi-head PEFT protocol. Across vision and language benchmarks, including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, TRISHUL improves convergence, stability, and final performance over federated LoRA baselines, with greater gains under stronger heterogeneity.
Comments18 pages, 17 figures, 11 tables
