arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1909.09706cs.LGcs.AIcs.ITmath.ITstat.ML
收藏

压缩表示法是否更好地泛化?

Do Compressed Representations Generalize Better?

Sharif University of Technology(谢里夫理工大学)

Hassan Hafez-Kolahi, Shohreh Kasaei, Mahdiyeh Soleymani-Baghshah

摘要:

机器学习中研究最多的问题之一是找到保证学习算法泛化的合理约束。这些约束通常表示为对目标的一些简单性假设。例如,在Vapnik-Chervonenkis(VC)理论中,可能假设的空间被认为具有有限的VC维度。在本文中,对输入变量$X$的熵$H(X)$的约束作为一个简单的假设进行了研究。证明了实现$ε$ - $δ$可能近似正确(PAC)假设的样本复杂度受$\frac{2^{ \left.6H(X)\middle/ε\right.}+\log{\frac{1}δ}}{ε^2}$的限制,该限制急剧上升到$\frac{1}{ε^2}$因子。此外,它表明,如果一个特征学习过程是用来学习压缩表示从数据集,这个界限不再存在。这些发现对信息瓶颈(IB)理论具有重要意义,该理论曾被用来解释深度神经网络(DNN)的泛化能力,但研究人员目前正在讨论其适用性。特别地,这是对先前的启发式的严格证明,即压缩表示以指数方式更容易学习。然而,我们的分析指出了两个因素,阻止了IB,在其目前的形式,适用于研究神经网络。首先,样本复杂度对$\frac{1}ε$的指数依赖性,当$ε$较小时,这会导致实际应用中边界的显著影响。其次,我们的分析表明,基于输入压缩的参数本质上不足以解释DNN等方法的泛化,其中特征也是使用可用数据学习的。

英文摘要:

One of the most studied problems in machine learning is finding reasonable constraints that guarantee the generalization of a learning algorithm. These constraints are usually expressed as some simplicity assumptions on the target. For instance, in the Vapnik-Chervonenkis (VC) theory the space of possible hypotheses is considered to have a limited VC dimension. In this paper, the constraint on the entropy $H(X)$ of the input variable $X$ is studied as a simplicity assumption. It is proven that the sample complexity to achieve an $ε$-$δ$ Probably Approximately Correct (PAC) hypothesis is bounded by $\frac{2^{ \left.6H(X)\middle/ε\right.}+\log{\frac{1}δ}}{ε^2}$ which is sharp up to the $\frac{1}{ε^2}$ factor. Morever, it is shown that if a feature learning process is employed to learn the compressed representation from the dataset, this bound no longer exists. These findings have important implications on the Information Bottleneck (IB) theory which had been utilized to explain the generalization power of Deep Neural Networks (DNNs), but its applicability for this purpose is currently under debate by researchers. In particular, this is a rigorous proof for the previous heuristic that compressed representations are exponentially easier to be learned. However, our analysis pinpoints two factors preventing the IB, in its current form, to be applicable in studying neural networks. Firstly, the exponential dependence of sample complexity on $\frac{1}ε$, which can lead to a dramatic effect on the bounds in practical applications when $ε$ is small. Secondly, our analysis reveals that arguments based on input compression are inherently insufficient to explain generalization of methods like DNNs in which the features are also learned using available data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑