arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.09075cs.CL
收藏

NLP中基于对比学习的对抗训练

Adversarial Training with Contrastive Learning in NLP

Handong Global University(韩东全球大学)

Daniela N. Rim, DongNyeong Heo, Heeyoul Choi

摘要:

多年来,对抗训练在自然语言处理(NLP)环境中得到了广泛的研究。主要目标是使模型鲁棒,以便相似的输入产生语义相似的结果,这不是一个微不足道的问题,因为语言中没有客观的语义相似性度量。以前的作品使用外部预训练的NLP模型来应对这一挑战,在训练过程中引入了额外的训练阶段,消耗了大量的内存。然而,最近流行的语言处理中的对比学习方法暗示了一种获得这种相似性限制的方便方法。对比学习方法的主要优点是,它的目标是将相似的数据点映射到彼此接近,并远离表示空间中的不同数据点。在这项工作中,我们提出了带有对比学习的对抗训练(ATCL),利用对比学习的好处来对抗性地训练语言处理任务。其核心思想是通过快速梯度法(FGM)在输入的嵌入空间中进行线性扰动,并通过对比学习训练模型以保持原始表示和扰动表示接近。在NLP实验中,我们将ATCL应用于语言建模和神经机器翻译任务。结果表明,与基线相比,ATCL不仅在定量(困惑度和BLEU)得分上有所改善,而且在不使用预训练模型的情况下,ATCL还在语义层面上实现了良好的定性结果。

英文摘要:

For years, adversarial training has been extensively studied in natural language processing (NLP) settings. The main goal is to make models robust so that similar inputs derive in semantically similar outcomes, which is not a trivial problem since there is no objective measure of semantic similarity in language. Previous works use an external pre-trained NLP model to tackle this challenge, introducing an extra training stage with huge memory consumption during training. However, the recent popular approach of contrastive learning in language processing hints a convenient way of obtaining such similarity restrictions. The main advantage of the contrastive learning approach is that it aims for similar data points to be mapped close to each other and further from different ones in the representation space. In this work, we propose adversarial training with contrastive learning (ATCL) to adversarially train a language processing task using the benefits of contrastive learning. The core idea is to make linear perturbations in the embedding space of the input via fast gradient methods (FGM) and train the model to keep the original and perturbed representations close via contrastive learning. In NLP experiments, we applied ATCL to language modeling and neural machine translation tasks. The results show not only an improvement in the quantitative (perplexity and BLEU) scores when compared to the baselines, but ATCL also achieves good qualitative results in the semantic level for both tasks without using a pre-trained model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑