arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.04579cs.LGcs.ITmath.ITstat.ML
收藏

可实现模型的信息论Bayes风险下界

Information-Theoretic Bayes Risk Lower Bounds for Realizable Models

Lily AI;Facebook AI(Facebook 人工智能研究院)

Matthew Nokleby, Ahmad Beirami

摘要:

我们推导出可实现的机器学习模型的贝叶斯风险和泛化误差的信息论下界。特别是,我们采用的分析,其中的率失真函数的模型参数的边界之间的训练样本和模型参数所需的互信息,以学习模型的贝叶斯风险约束。对于可实现的模型,我们表明,率失真函数和互信息承认表达式,便于分析。对于参数(大致)较低的Lipschitz模型,我们从下面限制率失真函数,而对于VC类,互信息由$d_\mathrm{vc}\log(n)$限制。当这些条件匹配时,关于0 - 1损失的贝叶斯风险的缩放速度不会快于$Ω(d_\mathrm{vc}/n)$,其匹配已知的外边界和最小最大下限直到对数因子。我们还考虑了标签噪声的影响,当训练和/或测试样本损坏时提供下限。

英文摘要:

We derive information-theoretic lower bounds on the Bayes risk and generalization error of realizable machine learning models. In particular, we employ an analysis in which the rate-distortion function of the model parameters bounds the required mutual information between the training samples and the model parameters in order to learn a model up to a Bayes risk constraint. For realizable models, we show that both the rate distortion function and mutual information admit expressions that are convenient for analysis. For models that are (roughly) lower Lipschitz in their parameters, we bound the rate distortion function from below, whereas for VC classes, the mutual information is bounded above by $d_\mathrm{vc}\log(n)$. When these conditions match, the Bayes risk with respect to the zero-one loss scales no faster than $Ω(d_\mathrm{vc}/n)$, which matches known outer bounds and minimax lower bounds up to logarithmic factors. We also consider the impact of label noise, providing lower bounds when training and/or test samples are corrupted.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑