arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03226cs.CLcs.AI
收藏

ATACompressor: 适应性任务感知压缩用于LLMs中高效长上下文处理

ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs

DCST, Tsinghua University(清华大学数据科学研究院)

Xuancheng Li, Haitao Li, Yujia Zhou, Qingyao Ai, Yiqun Liu

AI总结:

ATACompressor通过自适应任务感知压缩提升LLMs长上下文处理效率,实现信息保留与压缩效率的平衡。

摘要:

在大型语言模型(LLMs)中,长上下文输入常常面临'中间丢失'问题,即由于长度过长,关键信息变得稀释或被忽略。上下文压缩方法旨在通过减少输入大小来解决这一问题,但现有方法在平衡信息保留与压缩效率方面存在困难。我们提出了自适应任务感知压缩器(ATACompressor),它根据任务的具体需求动态调整压缩。ATACompressor采用选择性编码器,仅压缩长上下文中与任务相关的内容部分,确保关键信息得以保留同时减少不必要的内容。其自适应分配控制器感知相关内容的长度,并相应调整压缩率,以优化资源利用。我们在三个QA数据集上评估了ATACompressor:HotpotQA、MSMARCO和SQUAD,结果显示其在压缩效率和任务表现方面均优于现有方法。我们的方法为LLMs中的长上下文处理提供了可扩展的解决方案。此外,我们进行了多种消融研究和分析实验,以更深入地理解ATACompressor的关键组件。

英文摘要:

Long-context inputs in large language models (LLMs) often suffer from the "lost in the middle" problem, where critical information becomes diluted or ignored due to excessive length. Context compression methods aim to address this by reducing input size, but existing approaches struggle with balancing information preservation and compression efficiency. We propose Adaptive Task-Aware Compressor (ATACompressor), which dynamically adjusts compression based on the specific requirements of the task. ATACompressor employs a selective encoder that compresses only the task-relevant portions of long contexts, ensuring that essential information is preserved while reducing unnecessary content. Its adaptive allocation controller perceives the length of relevant content and adjusts the compression rate accordingly, optimizing resource utilization. We evaluate ATACompressor on three QA datasets: HotpotQA, MSMARCO, and SQUAD-showing that it outperforms existing methods in terms of both compression efficiency and task performance. Our approach provides a scalable solution for long-context processing in LLMs. Furthermore, we perform a range of ablation studies and analysis experiments to gain deeper insights into the key components of ATACompressor.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑