Matryoshka-Adaptor：面向更小嵌入维度的无监督与有监督微调
Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions
AI总结：
针对LLM高维嵌入计算延迟与成本高的问题，提出Matryoshka-Adaptor微调框架，兼容黑盒API，在无监督和有监督设置下实现2至12倍降维且不损失性能。
摘要：
来自大型语言模型（LLMs）的嵌入已成为各种应用中的关键组件，特别是在信息检索方面。虽然高维嵌入通常表现出更优越的性能，因为它们包含更多显著信息，但其实际应用经常受到计算延迟增加和相关更高成本的阻碍。为了应对这些挑战，我们提出了Matryoshka-Adaptor，一种专为定制LLM嵌入而设计的新型微调框架。Matryoshka-Adaptor促进了大幅度的降维，同时保持可比的性能水平，从而在计算效率和成本效益上实现了显著提升。我们的框架直接修改来自预训练LLMs的嵌入，其设计旨在与任何LLM架构无缝集成，包括仅能通过黑盒API访问的模型。此外，它在无监督和有监督学习环境中均表现出有效性。在涵盖英语、多语言和多模态数据集的多样化语料库上进行的严格评估一致表明，使用Matryoshka-Adaptor获得了显著收益。值得注意的是，借助Google和OpenAI Embedding API，Matryoshka-Adaptor实现了两到十二倍的降维，且在多个BEIR数据集上不妥协性能。
英文摘要：
Embeddings from Large Language Models (LLMs) have emerged as critical components in various applications, particularly for information retrieval. While high-dimensional embeddings generally demonstrate superior performance as they contain more salient information, their practical application is frequently hindered by elevated computational latency and the associated higher cost. To address these challenges, we propose Matryoshka-Adaptor, a novel tuning framework designed for the customization of LLM embeddings. Matryoshka-Adaptor facilitates substantial dimensionality reduction while maintaining comparable performance levels, thereby achieving a significant enhancement in computational efficiency and cost-effectiveness. Our framework directly modifies the embeddings from pre-trained LLMs which is designed to be seamlessly integrated with any LLM architecture, encompassing those accessible exclusively through black-box APIs. Also, it exhibits efficacy in both unsupervised and supervised learning settings. A rigorous evaluation conducted across a diverse corpus of English, multilingual, and multimodal datasets consistently reveals substantial gains with Matryoshka-Adaptor. Notably, with Google and OpenAI Embedding APIs, Matryoshka-Adaptor achieves a reduction in dimensionality ranging from two- to twelve-fold without compromising performance across multiple BEIR datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
