arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.05666cs.LGcs.IRstat.ML
收藏

概率分布学习及其在深度学习中的应用

Probability Distribution Learning and Its Application in Deep Learning

IEEE Publication Technology Group(IEEE出版技术组)

Binchuan Qi, Wei Gong, Li Li

AI总结:

本文提出概率分布学习框架,从理论上解释深度学习的优化与泛化机制,证明Fenchel-Young损失的必然性,引入广义凸性与光滑性,并推导模型无关的风险界,揭示关键因素影响。

摘要:

尽管深度学习在经验上取得了成功,但其在模型拟合和泛化方面仍缺乏全面的理论理解。本文提出了概率分布(PD)学习框架,以分析深度学习的优化和泛化机制。在该框架内,给定特征条件下标签的条件分布是主要的学习目标,损失函数、先验知识和模型属性被明确刻画。在这些公式下,我们建立了即使在非凸设置下也可优化性的理论保证,并推导出能为实际性能提供有意义解释的泛化误差界。具体而言,我们首先从理论上证明Fenchel-Young损失是解决PD学习问题的自然且必要的选择,从而论证了基于该损失所得结论的普适性。其次,为了捕捉深度神经网络(DNN)的特征,我们引入了$\mathcal{H}(ψ)$-凸性和$\mathcal{H}(Ψ)$-光滑性的概念,这些概念推广了经典强凸性和Lipschitz光滑性。基于这些概念,我们为SGD在训练DNN中的有效性提供了理论解释。最后,我们推导了训练模型期望风险与泛化误差的模型无关界,揭示了训练集大小、正则化项、标签与特征之间的互信息以及模型不可逆性造成的信息损失对风险和泛化的影响。基于我们的理论分析和实验验证,我们相信PD学习框架有助于对深度学习形成更深入、更统一的理论理解。

英文摘要:

Despite its empirical success, deep learning still lacks a comprehensive theoretical understanding of model fitting and generalization. This paper proposes the probability distribution (PD) learning framework to analyze the optimization and generalization mechanisms of deep learning. Within this framework, the conditional distribution of labels given features is the primary learning target, with the loss function, prior knowledge, and model properties explicitly characterized. Under these formulations, we establish theoretical guarantees on optimizability, even in non-convex settings, and derive generalization error bounds that provide meaningful explanations for practical performance. Specifically, we first prove theoretically that the Fenchel-Young loss is the natural and necessary choice for solving PD learning problems, thereby justifying the generality of conclusions based on this loss. Second, to capture the characteristics of deep neural networks (DNNs), we introduce the notions of $\mathcal{H}(ψ)$-convexity and $\mathcal{H}(Ψ)$-smoothness, which generalize the classical concepts of strong convexity and Lipschitz smoothness. Based on them, we provide a theoretical explanation for the effectiveness of SGD in training DNNs. Finally, we derive model-independent bounds on the expected risk and generalization error for trained models, revealing the influence of the training set size, regularization term, the mutual information between labels and features, and the information loss caused by model irreversibility on risk and generalization. Based on our theoretical analysis and experimental validation, we believe that the PD learning framework facilitates a deeper and more unified theoretical understanding of deep learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑