arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11573cs.CL
收藏

BGE Landmark Embedding:一种用于检索增强长上下文大型语言模型的无分块嵌入方法

BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models

Beijing Academy of Artificial Intelligence(北京人工智能研究院);Chinese Academy of Sciences(中国科学院)

Kun Luo, Zheng Liu, Shitao Xiao, Kang Liu

AI总结:

针对LLM上下文扩展成本高且质量差的问题,提出Extensible Embedding方法,用高信息密度的可扩展上下文嵌入替代单token嵌入,实现灵活、高效且兼容的上下文扩展。

摘要:

大型语言模型(LLMs)需要扩展上下文以处理许多关键应用。然而,现有方法面临上下文扩展成本高昂和质量不佳的问题。在本研究中,我们提出了Extensible Embedding(可扩展嵌入),以高灵活性和成本效益实现LLM上下文的高质量扩展。可扩展嵌入作为典型token embedding的增强,它表示可扩展上下文范围的信息,而非单个token。通过利用这种具有更高信息密度的的紧凑输入单元,LLM即使在小上下文窗口下也能访问广阔的上下文。可扩展嵌入在架构和训练方法上进行了系统性优化,1)上下文扩展的高灵活性,灵活支持多样上下文长度的临时扩展。2)训练的强样本效率,使嵌入模型能以高性价比方式学习。3)与现有LLMs的卓越兼容性,可扩展嵌入可作为插件组件无缝引入。在长上下文语言建模和理解任务上的全面评估验证,可扩展嵌入是扩展LLM上下文的有效、高效、灵活且兼容的方法。

英文摘要:

Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality of context extension. In this work, we proposeExtensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness. Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token. By leveraging such compact input units of higher information density, the LLM can access to a vast scope of context even with a small context window. Extensible embedding is systematically optimized in architecture and training method, which leads to multiple advantages. 1) High flexibility of context extension, which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way. 3) Superior compatibility with the existing LLMs, where the extensible embedding can be seamlessly introduced as a plug-in component. Comprehensive evaluations on long-context language modeling and understanding tasks verify extensible embedding as an effective, efficient, flexible, and compatible method to extend the LLM's context.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑