arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.04567cs.LGcs.IR
收藏

信息论视角下监督分类的误差界

Error Bounds of Supervised Classification from Information-Theoretic Perspective

College of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院)

Binchuan Qi

AI总结:

该研究从信息论视角分解监督分类的经验风险,推导拟合误差与泛化误差的界,进而得到期望风险上界,可解释深度学习多种现象,且实验验证了理论界与实际风险的正相关性。

摘要:

本文从信息论视角探究深度神经网络用于监督分类时期望风险的界。首先,我们引入模型风险与拟合误差,二者由对经验风险的进一步分解得到:模型风险表示基于模型预测概率的损失期望值,仅依赖于模型本身;拟合误差衡量经验风险与模型风险之间的差异。随后,我们推导了拟合误差的上界,该上界将反向传播的梯度、模型参数量与拟合误差关联起来。此外,我们证明泛化误差可由分类不确定性界定,而分类不确定性由分布的平滑性与样本规模共同表征。基于拟合误差与泛化的界,我们利用三角不等式建立了期望风险的上界,该界可用于为深度学习中的过参数化、非凸优化以及平坦极小值现象提供理论解释。最后,实验验证表明推导得到的理论界与实际期望风险之间存在显著的正相关性,从而证实了理论发现的实际相关性。

英文摘要:

In this paper, we explore bounds on the expected risk when using deep neural networks for supervised classification from an information theoretic perspective. Firstly, we introduce model risk and fitting error, which are derived from further decomposing the empirical risk. Model risk represents the expected value of the loss under the model's predicted probabilities and is exclusively dependent on the model. Fitting error measures the disparity between the empirical risk and model risk. Then, we derive the upper bound on fitting error, which links the back-propagated gradient and the model's parameter count with the fitting error. Furthermore, we demonstrate that the generalization errors are bounded by the classification uncertainty, which is characterized by both the smoothness of the distribution and the sample size. Based on the bounds on fitting error and generalization, by utilizing the triangle inequality, we establish an upper bound on the expected risk. This bound is applied to provide theoretical explanations for overparameterization, non-convex optimization and flat minima in deep learning. Finally, empirical verification confirms a significant positive correlation between the derived theoretical bounds and the practical expected risk, thereby affirming the practical relevance of the theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑