arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.00690cs.CL
收藏

利用对比微调改进较小语言模型的文本嵌入

Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning

Tsinghua University(清华大学)

Trapoom Ukarapol, Zhicheng Lee, Amy Xin

AI总结:

本文针对较小语言模型嵌入质量不足的问题,对MiniCPM、Phi-2和Gemma在NLI数据集上进行对比微调,在多个基准上提升文本嵌入质量,其中MiniCPM平均性能提升56.33%。

摘要:

尽管大型语言模型在自然语言理解中表现出卓越性能,但其资源密集型特性使它们较难被广泛使用。相比之下,MiniCPM等较小语言模型具有更可持续的可扩展性,但在缺乏专门优化时往往表现不佳。本文探索通过改进较小语言模型的文本嵌入来增强其能力。我们选取MiniCPM、Phi-2和Gemma三个语言模型,在NLI数据集上进行对比微调。结果表明,该微调方法在多个基准上提升了这三个模型的文本嵌入质量,其中MiniCPM改进最为显著,平均性能提升达56.33%。对比微调代码已公开于https://github.com/trapoom555/Language-Model-STS-CFT。

英文摘要:

While Large Language Models show remarkable performance in natural language understanding, their resource-intensive nature makes them less accessible. In contrast, smaller language models such as MiniCPM offer more sustainable scalability, but often underperform without specialized optimization. In this paper, we explore the enhancement of smaller language models through the improvement of their text embeddings. We select three language models, MiniCPM, Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our results demonstrate that this fine-tuning method enhances the quality of text embeddings for all three models across various benchmarks, with MiniCPM showing the most significant improvements of an average 56.33% performance gain. The contrastive fine-tuning code is publicly available at https://github.com/trapoom555/Language-Model-STS-CFT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑