arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.15078cs.CLcs.AI
收藏

不要从字面上理解:文本生成的编辑不变序列丢失

Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generation

Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Carnegie Mellon University(卡内基梅隆大学);Tsinghua University(清华大学);Sun Yat-Sen University(中山大学);JD AI Research(京东人工智能研究院);UC San Diego(加州大学圣地亚哥分校)

Guangyi Liu, Zichao Yang, Tianhua Tao, Xiaodan Liang, Junwei Bao, Zhen Li, Xiaodong He, Shuguang Cui, Zhiting Hu

摘要:

神经文本生成模型通常通过最大化具有序列交叉熵(CE)损失的对数似然来训练,这鼓励目标序列与生成的序列之间的精确令牌匹配。当靶序列不完美时,例如,当目标序列被噪声破坏时,或者当只有弱序列监督可用时。为了解决这个问题,我们提出了一种新的编辑不变序列损失(EISL),它计算目标n-gram与生成序列中所有n-gram的匹配损失。EISL被设计为对靶序列中的各种噪声和编辑具有鲁棒性。此外,EISL计算本质上是以目标n-gram为内核的近似卷积运算,这易于实现,并且使用现有库进行有效计算。为了证明EISL的有效性,我们进行了广泛的任务,包括机器翻译与噪声的目标序列,无监督的文本风格转移,只有弱的训练信号,和非自回归生成与非预定义的生成顺序的实验。实验结果表明,我们的方法显着优于常见的CE损失和其他强基线的所有任务。EISL有一个简单的API,可以用作CE丢失的直接替代:https://github.com/guangyliu/EISL。

英文摘要:

Neural text generation models are typically trained by maximizing log-likelihood with the sequence cross entropy (CE) loss, which encourages an exact token-by-token match between a target sequence with a generated sequence. Such training objective is sub-optimal when the target sequence is not perfect, e.g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available. To address the challenge, we propose a novel Edit-Invariant Sequence Loss (EISL), which computes the matching loss of a target n-gram with all n-grams in the generated sequence. EISL is designed to be robust to various noises and edits in the target sequences. Moreover, the EISL computation is essentially an approximate convolution operation with target n-grams as kernels, which is easy to implement and efficient to compute with existing libraries. To demonstrate the effectiveness of EISL, we conduct experiments on a wide range of tasks, including machine translation with noisy target sequences, unsupervised text style transfer with only weak training signals, and non-autoregressive generation with non-predefined generation order. Experimental results show our method significantly outperforms the common CE loss and other strong baselines on all the tasks. EISL has a simple API that can be used as a drop-in replacement of the CE loss: https://github.com/guangyliu/EISL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑