注意你的头：组装投影头以保障联邦模型的可靠性
Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Models
AI总结：
针对联邦模型在异构数据下因有偏投影头导致的校准差和不确定性低问题，提出组装投影头方法 APH，通过先验采样、差异化微调和头平均引入参数多样性，以不到30%的额外计算成本提升模型可靠性。
摘要：
联邦学习在异构数据下面临重大挑战，导致性能下降和收敛问题。尽管在缓解此类影响方面已取得相当进展，但联邦模型的可靠性方面在很大程度上被忽视。在本研究中，我们进行了大量实验，以研究通用联邦模型和个性化联邦模型的可靠性。我们的探索揭示了一个重要发现：联邦模型在面对异构数据时表现出不可靠性，在分布内测试数据上校准效果差，在分布外数据上不确定性水平低。这种不可靠性主要归因于有偏的投影头的存在，这些投影头将错误校准引入了联邦模型。受此观察启发，我们提出了“组装投影头”（APH）方法，以增强联邦模型的可靠性。通过将现有投影头参数视为先验，APH 从先验中随机采样多个投影头的初始化参数，并在不同学习率下对本地可用数据进行有针对性的微调。这种头集成将参数多样性引入确定性模型，消除了偏差，并通过头平均产生可靠的预测。我们在三个著名的联邦基准上评估了所提出的 APH 方法的有效性。实验结果验证了 APH 在模型校准和不确定性估计方面的功效。值得注意的是，APH 可以无缝集成到各种联邦方法中，但在大型模型中进行 100 倍推理时，仅需要不到 30% 的额外计算成本。
英文摘要：
Federated learning encounters substantial challenges with heterogeneous data, leading to performance degradation and convergence issues. While considerable progress has been achieved in mitigating such an impact, the reliability aspect of federated models has been largely disregarded. In this study, we conduct extensive experiments to investigate the reliability of both generic and personalized federated models. Our exploration uncovers a significant finding: \textbf{federated models exhibit unreliability when faced with heterogeneous data}, demonstrating poor calibration on in-distribution test data and low uncertainty levels on out-of-distribution data. This unreliability is primarily attributed to the presence of biased projection heads, which introduce miscalibration into the federated models. Inspired by this observation, we propose the "Assembled Projection Heads" (APH) method for enhancing the reliability of federated models. By treating the existing projection head parameters as priors, APH randomly samples multiple initialized parameters of projection heads from the prior and further performs targeted fine-tuning on locally available data under varying learning rates. Such a head ensemble introduces parameter diversity into the deterministic model, eliminating the bias and producing reliable predictions via head averaging. We evaluate the effectiveness of the proposed APH method across three prominent federated benchmarks. Experimental results validate the efficacy of APH in model calibration and uncertainty estimation. Notably, APH can be seamlessly integrated into various federated approaches but only requires less than 30\% additional computation cost for 100$\times$ inferences within large models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
