arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.16829cs.LGcs.CL
收藏

GISTEmbed:面向文本嵌入微调的训练负样本引导式批内选择

GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning

The World Bank(世界银行)

Aivin V. Solatorio

AI总结:

GISTEmbed提出利用引导模型改进对比训练中的批内负样本选择,减少噪声并提升嵌入模型微调性能,在MTEB上取得领先结果,并借助大模型能力增强小模型。

摘要:

嵌入模型是语义搜索、个性化推荐以及面向大语言模型的检索增强生成等人工智能应用的核心组成部分,因此需要高质量的训练数据。然而,人工数据整理的可扩展性有限,促使人们需要自动化方法来保证数据完整性。传统的无监督三元组挖掘能够自动生成训练数据,对嵌入模型训练至关重要,但会在无意中引入偏差和噪声,从而降低模型性能。针对这一问题,我们提出了GISTEmbed,这是一种通过引导模型在对比训练期间增强批内负样本选择的新策略。该方法不再依赖随机采样和批内负样本效用相等的假设,显著减少了由数据质量问题引起的噪声,并改善了模型微调效果。在Massive Text Embedding Benchmark(MTEB)上进行的基准测试表明,GISTEmbed在不同模型规模上均展现出持续的性能提升,并在部分类别中取得了最先进的结果。该框架通过利用功能强大但资源密集的大型模型的能力,使较小模型能够获得显著增强。GISTEmbed有望彻底改变高效小型模型的创建方式,使先进人工智能技术的获取更加民主化。让这些技术更加易用且具有成本效益,尤其是对于资源受限的应用而言,将显著扩大最先进人工智能解决方案在各行各业的影响力和可及性。

英文摘要:

Embedding models are integral to AI applications like semantic search, personalized recommendations, and retrieval augmented generation for LLMs, necessitating high-quality training data. However, the limited scalability of manual data curation prompts the need for automated methods to ensure data integrity. Traditional unsupervised triplet mining automates training data generation, crucial for embedding model training, yet inadvertently injects biases and noise, thereby degrading model performance. Addressing this, we introduce GISTEmbed, a novel strategy that enhances in-batch negative selection during contrastive training through a guide model. This approach departs from reliance on random sampling and equal utility assumption of batch negatives, significantly reducing noise from data quality issues and improving model fine-tuning. Benchmarked against the Massive Text Embedding Benchmark (MTEB), GISTEmbed showcases consistent performance improvements across various model sizes and achieves state-of-the-art results in select categories. This framework enables significant enhancements for smaller models by leveraging the capabilities of powerful yet resource-intensive large models. GISTEmbed can potentially revolutionize the creation of highly efficient, smaller models, democratizing access to advanced AI technologies. Making these technologies more accessible and cost-effective, especially for applications constrained by resources, significantly expands the impact and accessibility of state-of-the-art AI solutions across diverse sectors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑