arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.08462cs.CL
收藏

应用SoftTriple损失进行有监督语言模型微调

Applying SoftTriple Loss for Supervised Language Model Fine Tuning

Warsaw University of Technology(华沙理工大学)

Witold Sosnowski, Anna Wroblewska, Piotr Gawrysiak

摘要:

我们引入了一个新的损失函数TripleEntropy,以提高基于交叉熵和SoftTriple损失的微调一般知识预训练语言模型的分类性能。该损失函数可以使交叉熵损失微调后的鲁棒RoBERTa基线模型提高约0.02% - 2.29%。对流行数据集的彻底测试表明稳定的增长。训练数据集中的样本越少,增益就越高,因此,对于小型数据集,增益为0.78%,对于中型数据集,增益为0.86%,对于大型数据集,增益为0.20%,对于超大型数据集,增益为0.04%。

英文摘要:

We introduce a new loss function TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models based on cross-entropy and SoftTriple loss. This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about (0.02% - 2.29%). Thorough tests on popular datasets indicate a steady gain. The fewer samples in the training dataset, the higher gain -- thus, for small-sized dataset it is 0.78%, for medium-sized -- 0.86% for large -- 0.20% and for extra-large 0.04%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑