F^2-Softmax:通过频率因子化Softmax实现多样化的神经文本生成
F^2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax
摘要:
尽管最近在神经文本生成方面取得了进展,但对人类语言丰富多样性的编码仍然是难以捉摸的。我们认为,次优的文本生成主要是由于不平衡的令牌分布,特别是误导学习模型时,训练的最大似然目标。作为一种简单而有效的补救措施,我们提出了两种新的方法,F^2-Softmax和MefMax,即使在倾斜的频率分布下也可以进行平衡训练。MefMax将令牌唯一地分配给频率类,尝试将具有相似频率的令牌分组,并在类之间均衡频率质量。然后,F^2-Softmax将目标令牌的概率分布分解为(i)频率类别和(ii)来自目标频率类别的令牌的两个条件概率的乘积。模型学习更均匀的概率分布,因为它们局限于词汇表的子集。在七个相关指标上的显着性能增益表明,我们的方法不仅在提高多样性方面,而且在提高生成文本的质量方面都是至高无上的。
英文摘要:
Despite recent advances in neural text generation, encoding the rich diversity in human language remains elusive. We argue that the sub-optimal text generation is mainly attributable to the imbalanced token distribution, which particularly misdirects the learning model when trained with the maximum-likelihood objective. As a simple yet effective remedy, we propose two novel methods, F^2-Softmax and MefMax, for a balanced training even with the skewed frequency distribution. MefMax assigns tokens uniquely to frequency classes, trying to group tokens with similar frequencies and equalize frequency mass between the classes. F^2-Softmax then decomposes a probability distribution of the target token into a product of two conditional probabilities of (i) frequency class, and (ii) token from the target frequency class. Models learn more uniform probability distributions because they are confined to subsets of vocabularies. Significant performance gains on seven relevant metrics suggest the supremacy of our approach in improving not only the diversity but also the quality of generated texts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。