语言处理中的对比预训练入门：方法、经验教训和前景
A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives
摘要：
现代自然语言处理（NLP）方法采用自监督预训练目标，如掩蔽语言建模，以提高各种应用程序任务的性能。这些预训练方法经常扩展为递归、对抗或语言属性掩蔽，最近还扩展为对比学习目标。对比自监督训练目标通过学习将增强图像的输入-输入对对比为相似或不相似，使最近在图像表示预训练中取得了成功。然而，在NLP中，自动创建文本输入增强仍然非常具有挑战性，因为单个令牌可以颠倒句子的含义。出于这个原因，一些对比NLP预训练方法使用来自度量学习和基于能量的模型的方法来对比输入-标签对，而不是输入-输入对。在这项调查中，我们总结了最近的自监督和监督对比NLP预训练方法，并描述了它们用于改进语言建模，少量或零样本学习，预训练数据效率和特定NLP最终任务的地方。我们介绍了关键的对比学习概念，从以前的研究和结构工程的应用和跨领域的关系。最后，我们指出了对比NLP面临的挑战和未来的发展方向，以鼓励对比NLP预训练更接近最近在图像表示预训练中取得的成功。
英文摘要：
Modern natural language processing (NLP) methods employ self-supervised pretraining objectives such as masked language modeling to boost the performance of various application tasks. These pretraining methods are frequently extended with recurrence, adversarial or linguistic property masking, and more recently with contrastive learning objectives. Contrastive self-supervised training objectives enabled recent successes in image representation pretraining by learning to contrast input-input pairs of augmented images as either similar or dissimilar. However, in NLP, automated creation of text input augmentations is still very challenging because a single token can invert the meaning of a sentence. For this reason, some contrastive NLP pretraining methods contrast over input-label pairs, rather than over input-input pairs, using methods from Metric Learning and Energy Based Models. In this survey, we summarize recent self-supervised and supervised contrastive NLP pretraining methods and describe where they are used to improve language modeling, few or zero-shot learning, pretraining data-efficiency and specific NLP end-tasks. We introduce key contrastive learning concepts with lessons learned from prior research and structure works by applications and cross-field relations. Finally, we point to open challenges and future directions for contrastive NLP to encourage bringing contrastive NLP pretraining closer to recent successes in image representation pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
