带宽预算下的联邦语言模型：蒸馏率与共形覆盖
Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage
AI总结：
本文研究带宽受限节点间分布式语言模型的统计保证，提出联邦探针-对数蒸馏（FPLD）和联邦共形RAG（FC-RAG）两种协议，分别给出训练时的KL一致性率和推理时的无分布边际覆盖界，首次将带宽作为一阶统计参数。
摘要：
在临床网络、企业知识库和科学联盟中，经常出现数据分散在带宽受限节点上且无法集中的场景，需要训练语言模型。我们研究数据必须保持分布式在节点上的情况，并询问在明确带宽预算下原则上可以实现哪些统计保证；我们的目标是描述可证明的可能性，而不是展示一个可部署的系统。现有理论要么单独处理训练时的一致性，要么单独处理推理时的校准，且没有先前的工作将带宽作为一阶统计参数。我们分析了两种协议：用于训练的联邦探针-对数蒸馏（FPLD）和用于推理的联邦共形RAG（FC-RAG），作为我们结果的分析载体。我们的第一个主要结果是FPLD的显式高概率KL一致性率，同时依赖于节点数$K$、每节点样本量$n$、量化预算$B$、探针集大小$m$和词汇量$V$；带宽仅通过指数衰减的量化项进入。我们的第二个主要结果是FC-RAG的无分布边际覆盖界，其新颖的检索带宽松弛量$\Delta_{\mathrm{RAG}} = f_{\max}\sqrt{K^{-2}\sum_i v(B_i)}$使每节点检索带宽成为一阶统计参数，在每节点均匀情况下，通过$K$个节点的算术聚合使松弛量以$K^{-1/2}$的速度缩小。一个Pinsker型推论将两个界组合成端到端的覆盖保证。合成实验验证了沿界参数的预测缩放；在GPT-2测试平台上的小规模实验表明，定性带宽-准确率权衡在真实语言模型上仍然存在。部署规模的实证评估超出范围。
英文摘要：
Hospitals, companies, and research consortia want language models that learn from, and answer questions over, data that must stay where it is held, over links with limited bandwidth, such as the satellite links of remote clinics and ships. Averaging the sites' model parameters makes each message grow with the model and requires one shared architecture; methods that exchange predictions avoid both limits, but, to our knowledge, none bounds what compressing those predictions costs. We introduce two methods that compress every message at a small and predictable cost. Federated Probe-Logit Distillation (FPLD) trains a shared language model: each site sends compressed logits on shared unlabeled probes, and a hub distills one model from their average. Federated Conformal Retrieval-Augmented Generation (FC-RAG) answers questions: each site scores candidate answers using its own documents and sends compressed scores, and the hub returns a conformal prediction set that contains the correct answer at a guaranteed rate. Both use one codec, which centers each message and rounds it after adding a random offset known to the receiver. For logits within a fixed clipping range, we bound the expected log-loss added by compression independently of vocabulary size; the bound depends only on the bits, the range, and the number of sites, falls as more sites take part, and lets the bit budget be set before deployment. We also prove that recalibrating after compression keeps the coverage guarantee, and that FPLD's training error bounds how far coverage can slip when a threshold is reused after a model update. In experiments with GPT-2 and Qwen2.5, 8 bits per number cause no detectable loss in the trained model or in coverage. FPLD also trains a shared model across different architectures, where parameter averaging does not apply; with one shared architecture, parameter averaging remains more accurate.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
