Extensible Embedding:LLM上下文长度的灵活乘数
Extensible Embedding: A Flexible Multipler For LLM's Context Length
AI总结:
针对LLM上下文扩展成本高且质量低的问题,提出Extensible Embedding作为高密度紧凑输入单元,以即插即用方式高效灵活地扩展上下文长度。
摘要:
大型语言模型(LLMs)需要扩展上下文以处理许多关键应用。然而,现有的方法面临上下文扩展成本高昂和质量欠佳的问题。在本研究中,我们提出了Extensible Embedding,它以极强的灵活性和成本效益实现了LLM上下文的高质量扩展。Extensible Embedding作为典型token embedding的增强,表示可扩展上下文范围的信息,而非单个token。通过利用这种信息密度更高的紧凑输入单元,LLM即使上下文窗口很小也能访问广阔的上下文范围。Extensible Embedding在架构和训练方法上进行了系统性优化,带来多重优势:1)上下文扩展的高灵活性,灵活支持不同上下文长度的临时扩展;2)训练的强样本效率,使embedding模型能以高性价比方式学习;3)与现有LLM的卓越兼容性,可作为即插即用组件无缝引入。在长上下文语言建模和理解任务上的全面评估验证了Extensible Embedding是一种扩展LLM上下文的有效、高效、灵活且兼容的方法。
英文摘要:
Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality of context extension. In this work, we propose Extensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness. Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token. By leveraging such compact input units of higher information density, the LLM can access to a vast scope of context even with a small context window. Extensible embedding is systematically optimized in architecture and training method, which leads to multiple advantages. 1) High flexibility of context extension, which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way. 3) Superior compatibility with the existing LLMs, where the extensible embedding can be seamlessly introduced as a plug-in component. Comprehensive evaluations on long-context language modeling and understanding tasks verify extensible embedding as an effective, efficient, flexible, and compatible method to extend the LLM's context.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。