arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2105.14849cs.LGcs.AIcs.CLcs.NEcs.SDeess.ASmath.STstat.TH
收藏

为什么CTC会导致峰值行为?

Why does CTC result in peaky behavior?

RWTH Aachen University(亚琛工业大学);AppTek GmbH(AppTek有限责任公司)

Albert Zeyer, Ralf Schlüter, Hermann Ney

摘要:

CTC模型的峰值行为在实验上是众所周知的。然而,关于为什么会出现峰值行为以及这是否是一个好的属性的理解缺失。我们提供了一个正式的峰值行为和梯度下降收敛性能的CTC损失和相关的训练标准的分析。我们的分析提供了一个深刻的理解为什么峰值行为发生,当它是次优的。在一个简单的例子,这应该是微不足道的学习任何模型,我们证明了一个前馈神经网络训练与CTC从统一的初始化收敛到峰值的行为与100%的出错率。我们的分析进一步解释了为什么CTC只与空白标签一起工作。我们进一步证明,峰值行为不会发生在其他相关的损失,包括标签先验模型,这提高了收敛性。

英文摘要:

The peaky behavior of CTC models is well known experimentally. However, an understanding about why peaky behavior occurs is missing, and whether this is a good property. We provide a formal analysis of the peaky behavior and gradient descent convergence properties of the CTC loss and related training criteria. Our analysis provides a deep understanding why peaky behavior occurs and when it is suboptimal. On a simple example which should be trivial to learn for any model, we prove that a feed-forward neural network trained with CTC from uniform initialization converges towards peaky behavior with a 100% error rate. Our analysis further explains why CTC only works well together with the blank label. We further demonstrate that peaky behavior does not occur on other related losses including a label prior model, and that this improves convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑