arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.14161cs.CL
收藏

用于微调大型预训练语言模型的有监督对比学习多目标优化

Supervised Contrastive Learning as Multi-Objective Optimization for Fine-Tuning Large Pre-trained Language Models

Université Internationale de Rabat(拉巴特国际大学);INRIA-Lorraine(法国国家信息与自动化研究所洛林分部)

Youness Moukafih, Mounir Ghogho, Kamel Smaili

摘要:

近年来,有监督的对比学习(SCL)在大多数分类任务中表现出了优异的性能。在SCL中,一个神经网络被训练来优化两个目标:在嵌入空间中将锚点和阳性样本拉在一起,以及将锚点从阴性样本中推开。然而,这两个不同的目标可能会发生冲突,需要在优化过程中进行权衡。在这项工作中,我们制定的SCL问题作为一个多目标优化问题的微调阶段的RoBERTa语言模型。利用两种方法来解决优化问题:(i)线性标量化(LS)方法,它最小化每任务损失的加权线性组合;以及(ii)精确帕累托最优(EPO)方法,它找到帕累托前沿与给定偏好向量的交集。我们在几个GLUE基准任务上评估了我们的方法,而不使用数据增强、存储体或生成对抗性示例。实验结果表明,所提出的学习策略显着优于一个强大的竞争对比学习基线

英文摘要:

Recently, Supervised Contrastive Learning (SCL) has been shown to achieve excellent performance in most classification tasks. In SCL, a neural network is trained to optimize two objectives: pull an anchor and positive samples together in the embedding space, and push the anchor apart from the negatives. However, these two different objectives may conflict, requiring trade-offs between them during optimization. In this work, we formulate the SCL problem as a Multi-Objective Optimization problem for the fine-tuning phase of RoBERTa language model. Two methods are utilized to solve the optimization problem: (i) the linear scalarization (LS) method, which minimizes a weighted linear combination of pertask losses; and (ii) the Exact Pareto Optimal (EPO) method which finds the intersection of the Pareto front with a given preference vector. We evaluate our approach on several GLUE benchmark tasks, without using data augmentations, memory banks, or generating adversarial examples. The empirical results show that the proposed learning strategy significantly outperforms a strong competitive contrastive learning baseline

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑