arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.02644cs.CL
收藏

稀疏文本生成

Sparse Text Generation

Instituto de Telecomunicações(电信研究所);Priberam Labs(Priberam实验室);Institute of Systems and Robotics(系统与机器人研究所);LUMLIS (Lisbon ELLIS Unit), Instituto Superior Técnico(里斯本ELLIS单位LUMLIS,高等技术学院);Unbabel(Unbabel公司);Lisbon, Portugal(葡萄牙里斯本)

Pedro Henrique Martins, Zita Marinho, André F. T. Martins

摘要:

当前最先进的文本生成器构建在强大的语言模型(如GPT-2)上,实现了令人印象深刻的性能。然而,为了避免退化的文本,它们需要通过温度参数或特别截断技术从修改的softmax中采样,如top- $k$或nucleus采样。这造成了训练和测试条件之间的不匹配。在本文中,我们使用最近推出的entmax变换训练和样本从一个原生的稀疏语言模型,避免这种不匹配。结果是文本生成器在流畅性和一致性、更少的重复和更接近人类文本的n-gram多样性方面具有良好的性能。为了评估我们的模型,我们提出了三个新的度量比较稀疏或截断分布:$ε$ -困惑,sparsemax分数,詹森-香农发散。在故事完成和对话生成方面的人工评估实验表明,entmax采样导致了更吸引人和连贯的故事和对话。

英文摘要:

Current state-of-the-art text generators build on powerful language models such as GPT-2, achieving impressive performance. However, to avoid degenerate text, they require sampling from a modified softmax, via temperature parameters or ad-hoc truncation techniques, as in top-$k$ or nucleus sampling. This creates a mismatch between training and testing conditions. In this paper, we use the recently introduced entmax transformation to train and sample from a natively sparse language model, avoiding this mismatch. The result is a text generator with favorable performance in terms of fluency and consistency, fewer repetitions, and n-gram diversity closer to human text. In order to evaluate our model, we propose three new metrics for comparing sparse or truncated distributions: $ε$-perplexity, sparsemax score, and Jensen-Shannon divergence. Human-evaluated experiments in story completion and dialogue generation show that entmax sampling leads to more engaging and coherent stories and conversations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑