arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.06417cs.CL
收藏

神经文本生成的对比框架

A Contrastive Framework for Neural Text Generation

University of Cambridge(剑桥大学);Tencent AI Lab(腾讯人工智能实验室);DeepMind(深度思维);The University of Hong Kong(香港大学)

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, Nigel Collier

摘要:

文本生成对于许多自然语言处理应用都是非常重要的。然而,神经语言模型的基于最大化的解码方法(例如束搜索)通常会导致退化的解决方案-生成的文本是不自然的,并且包含不期望的重复。现有的方法通过采样或修改训练目标来引入随机性以降低某些令牌的概率(例如,不可能训练)。然而,它们往往导致缺乏连贯性的解决办法。在这项工作中,我们表明,模型退化的一个根本原因是令牌表示的各向异性分布。我们提出了一个对比解决方案:(i)SimCTG,一个对比训练目标,用于校准模型的表示空间,以及(ii)一种解码方法-对比搜索-鼓励多样性,同时保持生成文本的连贯性。广泛的实验和分析,从两种语言的三个基准测试表明,我们提出的方法显着优于目前最先进的文本生成方法的评估人类和自动度量。

英文摘要:

Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑