arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.05889cs.ITcs.LGmath.ITstat.ML
收藏

离散数据私有算法泛化误差的上界

Upper Bounds on the Generalization Error of Private Algorithms for Discrete Data

KTH Royal Institute of Technology(瑞典皇家理工学院);Ericsson Research(爱立信研究院)

Borja Rodríguez-Gálvez, Germán Bassi, Mikael Skoglund

摘要:

在这项工作中,我们从信息理论的角度研究算法的泛化能力。已经表明,算法的预期泛化误差由算法的输出假设的条件概率分布(给定训练数据集)与其边际概率分布之间的相对熵的函数从上而下限定。我们建立在这个事实,并引入一个数学公式,以获得这个相对熵的上限。假设数据是离散的,然后我们开发了一个策略,使用这个公式,基于类型和典型性的方法,找到稳定算法的泛化误差的显式上界,即,在给定类似输入数据集的情况下产生类似输出假设的算法。特别是,我们展示了这种策略的情况下,$ε$ -DP和$μ$ -GDP算法的界限。

英文摘要:

In this work, we study the generalization capability of algorithms from an information-theoretic perspective. It has been shown that the expected generalization error of an algorithm is bounded from above by a function of the relative entropy between the conditional probability distribution of the algorithm's output hypothesis, given the dataset with which it was trained, and its marginal probability distribution. We build upon this fact and introduce a mathematical formulation to obtain upper bounds on this relative entropy. Assuming that the data is discrete, we then develop a strategy using this formulation, based on the method of types and typicality, to find explicit upper bounds on the generalization error of stable algorithms, i.e., algorithms that produce similar output hypotheses given similar input datasets. In particular, we show the bounds obtained with this strategy for the case of $ε$-DP and $μ$-GDP algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑