arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.11481cs.CLcs.AI
收藏

基于相似性的标签平滑算法在对话生成中的应用

Similarity Based Label Smoothing For Dialogue Generation

University at Buffalo(纽约州立大学布法罗分校)

Sougata Saha, Souvik Das, Rohini Srihari

摘要:

生成式神经会话系统通常以最小化训练“硬”目标和预测逻辑之间的熵损失为目标进行训练。通常,性能增益和改进的泛化可以通过使用正则化技术来实现,如标签平滑,它将训练“硬”目标转换为“软”目标。然而,标签平滑在不正确的训练目标上强制执行数据独立的均匀分布,这导致对于每个正确目标的等概率不正确目标的不正确假设。在本文中,我们提出并实验将数据依赖的字相似性为基础的加权方法,不正确的目标概率在标签平滑的均匀分布,一个更自然的分布的基础上的语义。我们引入超参数来控制不正确的目标分布,并在两个标准的开放域对话语料库上报告了使用标准标签平滑损失训练的网络的显着性能增益。

英文摘要:

Generative neural conversational systems are generally trained with the objective of minimizing the entropy loss between the training "hard" targets and the predicted logits. Often, performance gains and improved generalization can be achieved by using regularization techniques like label smoothing, which converts the training "hard" targets to "soft" targets. However, label smoothing enforces a data independent uniform distribution on the incorrect training targets, which leads to an incorrect assumption of equi-probable incorrect targets for each correct target. In this paper we propose and experiment with incorporating data dependent word similarity based weighing methods to transforms the uniform distribution of the incorrect target probabilities in label smoothing, to a more natural distribution based on semantics. We introduce hyperparameters to control the incorrect target distribution, and report significant performance gains over networks trained using standard label smoothing based loss, on two standard open domain dialogue corpora.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑