应用SoftTriple损失进行有监督语言模型微调
Applying SoftTriple Loss for Supervised Language Model Fine Tuning
摘要:
我们引入了一个新的损失函数TripleEntropy,以提高基于交叉熵和SoftTriple损失的微调一般知识预训练语言模型的分类性能。该损失函数可以使交叉熵损失微调后的鲁棒RoBERTa基线模型提高约0.02% - 2.29%。对流行数据集的彻底测试表明稳定的增长。训练数据集中的样本越少,增益就越高,因此,对于小型数据集,增益为0.78%,对于中型数据集,增益为0.86%,对于大型数据集,增益为0.20%,对于超大型数据集,增益为0.04%。
英文摘要:
We introduce a new loss function TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models based on cross-entropy and SoftTriple loss. This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about (0.02% - 2.29%). Thorough tests on popular datasets indicate a steady gain. The fewer samples in the training dataset, the higher gain -- thus, for small-sized dataset it is 0.78%, for medium-sized -- 0.86% for large -- 0.20% and for extra-large 0.04%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。