COMI：通过边际信息增益的粗到细上下文压缩
COMI: Coarse-to-fine Context Compression via Marginal Information Gain
AI总结：
COMI通过边际信息增益实现粗到细的上下文压缩，在长上下文任务中提升压缩效率和语义保留能力。
摘要：
大型语言模型（LLMs）在多样任务上展示了卓越的能力。然而，它们在长上下文场景中的部署仍受计算效率低下和信息冗余的阻碍。上下文压缩方法通过显著减少输入长度并消除冗余来解决这些挑战。我们提出了COMI，一种粗到细的自适应上下文压缩框架，该框架在高压缩率下联合优化语义相关性和多样性。我们引入了边际信息增益（MIG），其定义为一个单位对输入查询的相关性减去与其他单位的语义冗余，指导压缩过程优先考虑相关且低冗余的信息。该框架分为两个阶段：（1）粗粒度组再分配，其中上下文被划分为组，并根据组间MIG动态分配压缩率，确保压缩预算与信息价值分布一致；（2）细粒度令牌合并，其中每个组内的令牌通过基于组内MIG的加权机制融合，从而在保留关键语义的同时避免冗余的积累。在多个任务（如问答（例如NaturalQuestions、2WikiMQA、HotpotQA和NarrativeQA）、总结（例如MultiNews）上使用各种骨干网络（例如LLaMA-2-7B、Qwen2-7B）的广泛实验表明，COMI在现有基线上大幅领先，例如在32倍压缩约束下，使用Qwen2-7B在NaturalQuestions上实现了约25点的精确匹配（EM）提升。
英文摘要：
Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse tasks. However, their deployment in long context scenarios remains hindered by computational inefficiency and information redundancy. Context compression methods address these challenges by significantly reducing input length and eliminating redundancy. We propose COMI, a coarse-to-fine adaptive context compression framework that jointly optimizes for semantic relevance and diversity under high compression rates. We introduce Marginal Information Gain (MIG), a metric defined as the relevance of a unit to the input query minus its semantic redundancy with other units, guiding the compression process to prioritize information that is both relevant and low redundant. The framework operates in two stages: (1) Coarse-Grained Group Reallocation, where the context is partitioned into groups and dynamically assigned compression rates based on inter-group MIG, ensuring compression budgets align with information value distribution; and (2) Fine-Grained Token Merging, where tokens within each group are fused via an intra-group MIG-based weighting mechanism, thereby preserving key semantics while avoiding the accumulation of redundancy. Extensive experiments across question-answering (e.g., NaturalQuestions, 2WikiMQA, HotpotQA and NarrativeQA), summarization (e.g., MultiNews) with various backbones (e.g., LLaMA-2-7B, Qwen2-7B) show that COMI outperforms existing baselines by a large margin, e.g., approximately 25-point Exact Match (EM) improvement under 32x compression constraint with Qwen2-7B on NaturalQuestions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
