arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2011.05231stat.MLcs.LG
收藏

交叉熵损失的使用和滥用:现代深度学习中的案例研究

Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning

Columbia University(哥伦比亚大学);Layer6 AI;Zuckerman Institute(扎克曼研究所)

Elliott Gordon-Rodriguez, Gabriel Loaiza-Ganem, Geoff Pleiss, John P. Cunningham

摘要:

现代深度学习主要是一门实验科学,其中经验的进步偶尔会以牺牲概率的严谨性为代价。在这里,我们专注于一个这样的例子;即使用分类交叉熵损失来建模不是严格分类的数据,而是在单纯形上取值。这种做法是标准的神经网络架构与标签平滑和演员模仿强化学习,等等。利用最近发现的连续分类分布,我们提出了这些模型的概率启发的替代方案,提供了一种更有原则和理论吸引力的方法。通过仔细的实验,包括消融研究,我们确定在这些模型中的性能优越的潜力,从而突出了一个适当的概率治疗的重要性,以及说明一些故障模式。

英文摘要:

Modern deep learning is primarily an experimental science, in which empirical advances occasionally come at the expense of probabilistic rigor. Here we focus on one such example; namely the use of the categorical cross-entropy loss to model data that is not strictly categorical, but rather takes values on the simplex. This practice is standard in neural network architectures with label smoothing and actor-mimic reinforcement learning, amongst others. Drawing on the recently discovered continuous-categorical distribution, we propose probabilistically-inspired alternatives to these models, providing an approach that is more principled and theoretically appealing. Through careful experimentation, including an ablation study, we identify the potential for outperformance in these models, thereby highlighting the importance of a proper probabilistic treatment, as well as illustrating some of the failure modes thereof.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑