arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.08193stat.MLcs.ITcs.LGmath.IT
收藏

最小通信成本统计学习

Minimal Communication-Cost Statistical Learning

Paris Research Center, Huawei Technologies France(华为技术法国巴黎研发中心);Université Gustave Eiffel, France(古斯塔夫·埃菲尔大学)

Milad Sefidgaran, Abdellatif Zaidi, Piotr Krasnowski

AI总结:

针对客户端需将学习模型低通信成本传至服务器的问题,提出联合训练与信源编码方案,通过 KL 散度约束同时保证训练损失、泛化误差和通信成本。

摘要:

一个可访问 $n$ 个训练数据样本的客户端设备需要获得一个统计假设或模型 $W$,随后将其发送到远程服务器。客户端与服务器设备共享某些公共随机序列以及假设空间上的先验。在此问题中,合适的假设或模型 $W$ 必须同时满足两个不同的设计准则:(i)推理阶段具有较小的(总体)风险;(ii)较小的“复杂度”,使其能够以最小通信成本传送给服务器。本文提出一种联合训练与信源编码方案,并给出可证明的期望意义保证,其中期望是对编码器输出消息求取的。具体而言,我们表明,通过对压缩学习模型 $\widehat{W}$ 在给定 $W$ 时所诱导的条件分布与先验之间适当的 Kullback-Leibler 散度施加约束,可以同时保证较小的平均经验风险(也称训练损失)、较小的平均泛化误差以及较小的平均通信成本。我们还考虑了一种单轮场景,其中对于编码器的每个输出消息,都能获得关于经验风险和泛化误差的保证。

英文摘要:

A client device which has access to $n$ training data samples needs to obtain a statistical hypothesis or model $W$ and then to send it to a remote server. The client and the server devices share some common randomness sequence as well as a prior on the hypothesis space. In this problem a suitable hypothesis or model $W$ should meet two distinct design criteria simultaneously: (i) small (population) risk during the inference phase and (ii) small 'complexity' for it to be conveyed to the server with minimum communication cost. In this paper, we propose a joint training and source coding scheme with provable in-expectation guarantees, where the expectation is over the encoder's output message. Specifically, we show that by imposing a constraint on a suitable Kullback-Leibler divergence between the conditional distribution induced by a compressed learning model $\widehat{W}$ given $W$ and the prior, one guarantees simultaneously small average empirical risk (aka training loss), small average generalization error and small average communication cost. We also consider a one-shot scenario in which the guarantees on the empirical risk and generalization error are obtained for every encoder's output message.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑