arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.17062cs.CL
收藏

UniICL:一个统一压缩、选择与生成的高效统一框架

UniICL: An Efficient Unified Framework Unifying Compression, Selection, and Generation

School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Hong Kong Polytechnic University(香港理工大学);Stepfun

Jun Gao, Qi Lv, Zili Wang, Tianxiang Wu, Ziqiang Cao, Wenjie Li

AI总结:

针对上下文学习因上下文过长和示例选择不当导致的效率与效果问题,提出统一框架UniICL,整合压缩、选择与生成,并通过示例库缓存避免重复压缩,经域外评估验证其有效性与高效性。

摘要:

上下文学习(ICL)通过在生成前添加少量示例来增强大型语言模型(LLMs)的推理能力。这促使研究者引入更多示例以提供额外的上下文信息。然而,现有方法因上下文长度过度增长而面临显著局限性,导致较大的硬件负担。此外,由现成工具选出的浅层相关示例阻碍了LLMs在生成过程中捕获有用的上下文信息。本文提出UniICL,一个新颖的统一ICL框架,将示例压缩、示例选择与最终响应生成统一起来。进一步地,为提升推理效率,我们设计了一种定制的压缩策略,使UniICL能将压缩结果缓存至“示例库”(DB),避免对相同示例的重复压缩。广泛的域外评估证明了UniICL在有效性和效率方面的优势。

英文摘要:

In-context learning (ICL) enhances the reasoning abilities of Large Language Models (LLMs) by prepending a few demonstrations. It motivates researchers to introduce more examples to provide additional contextual information for the generation. However, existing methods show a significant limitation due to the problem of excessive growth in context length, which causes a large hardware burden. In addition, shallow-relevant examples selected by off-the-shelf tools hinder LLMs from capturing useful contextual information for generation. In this paper, we propose \textbf{UniICL}, a novel \textbf{Uni}fied \textbf{ICL} framework that unifies demonstration compression, demonstration selection, and final response generation. Furthermore, to boost inference efficiency, we design a tailored compression strategy that allows UniICL to cache compression results into \textbf{Demonstration Bank} (\textbf{DB}), which avoids repeated compression of the same demonstration. Extensive out-of-domain evaluations prove the advantages of UniICL in both effectiveness and efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑