小数据集上音素表示学习的预测编码模型分析
Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets
摘要：
从人类语言习得的计算建模的角度来看，使用预测编码的神经网络模型是有趣的，其中目标是了解如何从没有任何标签的语音中学习语言单元。尽管在文献中已经提出了几种有前途的基于预测编码的学习算法，但目前还不清楚它们对不同语言和训练数据集大小的推广效果如何。此外，尽管这样的模型已被证明是有效的音素特征学习器，目前还不清楚是否最小化这些模型的预测损失函数也导致最佳的音素样表示。本研究探讨了两种预测编码模型的行为，自回归预测编码和对比预测编码，在音素辨别任务（ABX任务）的两种语言具有不同的数据集大小。我们的实验表明，与两个数据集的自回归损失和音素歧视分数之间的强相关性。然而，令我们惊讶的是，CPC模型在经过一次训练数据之后就已经显示出快速收敛，并且平均而言，其表示在两种语言上都优于APC。
英文摘要：
Neural network models using predictive coding are interesting from the viewpoint of computational modelling of human language acquisition, where the objective is to understand how linguistic units could be learned from speech without any labels. Even though several promising predictive coding -based learning algorithms have been proposed in the literature, it is currently unclear how well they generalise to different languages and training dataset sizes. In addition, despite that such models have shown to be effective phonemic feature learners, it is unclear whether minimisation of the predictive loss functions of these models also leads to optimal phoneme-like representations. The present study investigates the behaviour of two predictive coding models, Autoregressive Predictive Coding and Contrastive Predictive Coding, in a phoneme discrimination task (ABX task) for two languages with different dataset sizes. Our experiments show a strong correlation between the autoregressive loss and the phoneme discrimination scores with the two datasets. However, to our surprise, the CPC model shows rapid convergence already after one pass over the training data, and, on average, its representations outperform those of APC on both languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
