ACL：对齐对比学习提升BERT和多退出BERT微调
ACL: Aligned Contrastive Learning Improves BERT and Multi-exit BERT Fine-tuning
AI总结：
ACL通过融合对比学习与交叉熵损失，提升BERT及多退出BERT的微调性能，实现更优的质量-速度平衡。
摘要：
尽管在自监督学习中取得了成功，对比学习在监督设置中研究较少。在本工作中，我们首先通过一系列试点实验表明，在监督设置中，交叉熵损失目标（CE）和对比学习目标往往相互冲突，从而阻碍了CL在监督设置中的应用。为了解决这个问题，我们引入了一种新的对齐对比学习（ACL）框架。首先，ACL-Embed将标签嵌入视为带有不同标签的额外增强样本，并利用对比学习将标签嵌入与其样本表示对齐。其次，为了促进ACL-Embed目标与CE损失的优化，我们提出了ACL-Grad，该方法会在两个目标冲突时丢弃ACL-Embed项。为进一步提升多退出BERT中间退出的性能，我们进一步提出了跨层ACL（ACL-CL），即让教师退出指导学生浅层退出的优化。在GLUE基准上的广泛实验结果得出以下结论：（a）ACL-BRT在GLUE任务中优于或与CE和CE+SCL相当；（b）ACL，尤其是CL-ACL，在多退出BERT微调中显著优于基线方法，从而为低延迟应用提供了更好的质量-速度权衡。
英文摘要：
Despite its success in self-supervised learning, contrastive learning is less studied in the supervised setting. In this work, we first use a set of pilot experiments to show that in the supervised setting, the cross-entropy loss objective (CE) and the contrastive learning objective often conflict with each other, thus hindering the applications of CL in supervised settings. To resolve this problem, we introduce a novel \underline{A}ligned \underline{C}ontrastive \underline{L}earning (ACL) framework. First, ACL-Embed regards label embeddings as extra augmented samples with different labels and employs contrastive learning to align the label embeddings with its samples' representations. Second, to facilitate the optimization of ACL-Embed objective combined with the CE loss, we propose ACL-Grad, which will discard the ACL-Embed term if the two objectives are in conflict. To further enhance the performances of intermediate exits of multi-exit BERT, we further propose cross-layer ACL (ACL-CL), which is to ask the teacher exit to guide the optimization of student shallow exits. Extensive experiments on the GLUE benchmark results in the following takeaways: (a) ACL-BRT outperforms or performs comparably with CE and CE+SCL on the GLUE tasks; (b) ACL, especially CL-ACL, significantly surpasses the baseline methods on the fine-tuning of multi-exit BERT, thus providing better quality-speed tradeoffs for low-latency applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
