纠错神经序列预测
Error-Correcting Neural Sequence Prediction
摘要：
我们提出了一种新的神经序列预测方法的基础上\textit{error-correcting output codes}，避免了精确的softmax归一化，并允许速度和性能之间的权衡。我们使用纠错码来表示预测和输出，而不是最小化预测概率分布和真实分布之间的度量。其次，我们提出了多种方法来提高准确性和收敛速度，通过最大限度地提高代码之间的可分性，对应于类成比例的词嵌入相似性。最后，我们介绍了我们的主要贡献，称为\textit{Latent Variable Mixture Sampling}，一种用于减轻暴露偏差的技术，它可以集成到训练基于潜在变量的神经序列预测器，如ECOC。这涉及以两种方式之一来混合过去预测和过去目标的潜在代码：（1）根据预定义的采样时间表或（2）可微分采样过程，其中通过用平滑近似替换贪婪argmax运算来在整个训练过程中学习混合概率。ECOC-NSP导致语言建模数据集的持续改进，并且发现所提出的潜变量混合采样方法对于文本生成任务（例如图像描述）表现良好。
英文摘要：
We propose a novel neural sequence prediction method based on \textit{error-correcting output codes} that avoids exact softmax normalization and allows for a tradeoff between speed and performance. Instead of minimizing measures between the predicted probability distribution and true distribution, we use error-correcting codes to represent both predictions and outputs. Secondly, we propose multiple ways to improve accuracy and convergence rates by maximizing the separability between codes that correspond to classes proportional to word embedding similarities. Lastly, we introduce our main contribution called \textit{Latent Variable Mixture Sampling}, a technique that is used to mitigate exposure bias, which can be integrated into training latent variable-based neural sequence predictors such as ECOC. This involves mixing the latent codes of past predictions and past targets in one of two ways: (1) according to a predefined sampling schedule or (2) a differentiable sampling procedure whereby the mixing probability is learned throughout training by replacing the greedy argmax operation with a smooth approximation. ECOC-NSP leads to consistent improvements on language modelling datasets and the proposed Latent Variable mixture sampling methods are found to perform well for text generation tasks such as image captioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
