arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.00460cs.CL
收藏

真否定句的语言模型预训练

Language Model Pre-training on True Negatives

Shanghai Jiao Tong University(上海交通大学);National Institute of Information and Communications Technology (NICT)(国立情报学研究所)

Zhuosheng Zhang, Hai Zhao, Masao Utiyama, Eiichiro Sumita

摘要:

区分性预训练的语言模型(PLM)学习从故意损坏的文本中预测原始文本。以前一个文本为正样本,后一个文本为负样本,可以有效地训练PLM进行语境化表示。然而,这种类型的PLM的训练高度依赖于自动构造的样本的质量。现有的PLM简单地将所有损坏的文本视为相等的否定而不进行任何检查,这实际上使所得到的模型不可避免地遭受假否定问题,其中在伪否定数据上进行训练,并导致所得到的PLM的效率和鲁棒性降低。本文在定义歧视性PLM中长期被忽视的假阴性问题的基础上,设计了增强的预训练方法来抵消假阴性预测,并通过纠正受假阴性预测影响的有害梯度更新,鼓励对真阴性的语言模型进行预训练。在GLUE和SQuAD基准测试上的实验结果表明,我们的反假阴性预训练方法确实带来了更好的性能以及更强的鲁棒性。

英文摘要:

Discriminative pre-trained language models (PLMs) learn to predict original texts from intentionally corrupted ones. Taking the former text as positive and the latter as negative samples, the PLM can be trained effectively for contextualized representation. However, the training of such a type of PLMs highly relies on the quality of the automatically constructed samples. Existing PLMs simply treat all corrupted texts as equal negative without any examination, which actually lets the resulting model inevitably suffer from the false negative issue where training is carried out on pseudo-negative data and leads to less efficiency and less robustness in the resulting PLMs. In this work, on the basis of defining the false negative issue in discriminative PLMs that has been ignored for a long time, we design enhanced pre-training methods to counteract false negative predictions and encourage pre-training language models on true negatives by correcting the harmful gradient updates subject to false negative predictions. Experimental results on GLUE and SQuAD benchmarks show that our counter-false-negative pre-training methods indeed bring about better performance together with stronger robustness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑