arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.08701cs.LG
收藏

对比学习能否优化嵌入表示

Can Contrastive Learning Refine Embeddings

University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Amazon(亚马逊公司)

Lihui Liu, Jinha Kim, Vidit Bansal

AI总结:

针对现有对比学习忽略预训练编码器输出潜力的问题,本文提出新型框架SIMSKIP,以编码器输出嵌入为输入进行优化,理论证明其不会增大下游任务误差上界,实验验证其可提升下游任务性能。

摘要:

对比学习的最新进展彻底革新了自监督表示学习,并在基准任务上取得了最优性能。现有方法大多聚焦于将对比学习应用于图像、自然语言句子或网络等输入数据模态,却忽略了利用预训练编码器输出的潜力。本文提出了SIMSKIP,这是一种专门为下游任务优化输入嵌入的新型对比学习框架。与传统无监督学习方法不同,SIMSKIP以编码器模型的输出嵌入作为输入。通过理论分析,我们证明应用SIMSKIP不会使下游任务误差的上界大于作为其输入的原始嵌入对应的误差上界。在多个公开数据集上的实验结果表明,SIMSKIP生成的嵌入能够提升下游任务的性能。

英文摘要:

Recent advancements in contrastive learning have revolutionized self-supervised representation learning and achieved state-of-the-art performance on benchmark tasks. While most existing methods focus on applying contrastive learning to input data modalities such as images, natural language sentences, or networks, they overlook the potential of utilizing outputs from previously trained encoders. In this paper, we introduce SIMSKIP, a novel contrastive learning framework that specifically refines input embeddings for downstream tasks. Unlike traditional unsupervised learning approaches, SIMSKIP takes advantage of the output embeddings of encoder models as its input. Through theoretical analysis, we provide evidence that applying SIMSKIP does not result in larger upper bounds on downstream task errors than those of the original embeddings, which serve as SIMSKIP's input. Experimental results on various open datasets demonstrate that the embeddings produced by SIMSKIP improve performance on downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑