使用UltraGist压缩长上下文
Compressing Lengthy Context With UltraGist
AI总结:
提出UltraGist方法,通过创新的压缩与学习算法实现灵活、细粒度且高效的长上下文压缩,在多种长上下文任务中保持近乎无损的压缩性能。
摘要:
压缩长上下文是一个关键但技术上具有挑战性的问题。在本文中,我们提出了一种名为UltraGist的新方法,由于压缩和学习算法的创新设计,该方法在长上下文的高质量压缩方面表现出色。UltraGist带来以下重要益处。首先,它显著提升了压缩的灵活性,因为它可以有效地学习以支持广泛的上下文长度和压缩比。其次,它有助于对长上下文产生细粒度的压缩,其中上下文的每个小片段都在定制的交叉注意力机制基础上逐步处理。第三,它使训练过程具有样本效率,从而最大化训练数据的使用。最后,它促进了动态上下文压缩的高效运行,因为压缩结果可以逐步生成并因此增量更新。UltraGist在与长上下文相关的各种任务上进行了评估,例如文档问答和摘要、少样本学习、多会话对话等。虽然现有方法无法处理这些具有挑战性的场景,但我们的方法能够在所有评估中保持近乎无损的压缩性能。我们的数据、模型和代码已在\url{https://github.com/namespace-Pt/UltraGist}发布。
英文摘要:
Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality compression of lengthy context due to the innovative design of the compression and learning algorithm. UltraGist brings forth the following important benefits. Firstly, it notably contributes to the flexibility of compression, as it can be effectively learned to support a broad range of context lengths and compression ratios. Secondly, it helps to produce fine-grained compression for the lengthy context, where each small segment of the context is progressively processed on top of a tailored cross-attention mechanism. Thirdly, it makes the training process sample-efficient and thus maximizes the use of training data. Finally, it facilitates the efficient running of compression for dynamic context, as the compression result can be progressively generated and hence incrementally updated. UltraGist is evaluated on a wide variety of tasks associated with lengthy context, such as document QA and summarization, few-shot learning, multi-session conversation, et al. Whilst the existing methods fail to handle these challenging scenarios, our approach is able to preserve a near-lossless compression performance throughout all the evaluations. Our data, model, and code have been released at \url{https://github.com/namespace-Pt/UltraGist}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。