arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1804.05474cs.LGcs.ITmath.ITstat.ML
收藏

学习信息复杂性的一个直和结果

A Direct Sum Result for the Information Complexity of Learning

Technion-IIT(以色列理工学院);University of California, Berkeley(加州大学伯克利分校)

Ido Nachum, Jonathan Shafer, Amir Yehudayoff

摘要:

PAC学习VC维$d$的一类假设需要多少比特的信息?我们遵循的数学设置是Basily等人(2018)的数学设置,其中感兴趣的值是输入样本$S$和学习算法$A$输出的假设之间的互信息$\mathrm{I}(S;A(S))$。本文在定义域$\mathcal{X}$上引入了一类VC维$d$函数,它的信息复杂度至少为$Ω\left(d\log \log \frac{|\mathcal{X}|}{d}\right)$ bits,对于任意的相容的和适当的算法(确定的或随机的). Basily等人证明了$d=1$情况下的类似(但定量上较弱)结果。上述结果实际上是我们所探讨的一个更普遍现象的一个特例。我们定义了一类给定函数的信息复杂性的概念$\mathcal{H}$。直观地说,它是$\mathcal{H}$算法必须保留的关于其输入的最小信息量,以确保一致性和正确性。在这种情况下,我们证明了信息复杂性的直接和结果,粗略地说,当结合几个类的信息复杂性总和。

英文摘要:

How many bits of information are required to PAC learn a class of hypotheses of VC dimension $d$? The mathematical setting we follow is that of Bassily et al. (2018), where the value of interest is the mutual information $\mathrm{I}(S;A(S))$ between the input sample $S$ and the hypothesis outputted by the learning algorithm $A$. We introduce a class of functions of VC dimension $d$ over the domain $\mathcal{X}$ with information complexity at least $Ω\left(d\log \log \frac{|\mathcal{X}|}{d}\right)$ bits for any consistent and proper algorithm (deterministic or random). Bassily et al. proved a similar (but quantitatively weaker) result for the case $d=1$. The above result is in fact a special case of a more general phenomenon we explore. We define the notion of information complexity of a given class of functions $\mathcal{H}$. Intuitively, it is the minimum amount of information that an algorithm for $\mathcal{H}$ must retain about its input to ensure consistency and properness. We prove a direct sum result for information complexity in this context; roughly speaking, the information complexity sums when combining several classes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑