超越位置偏差:从位置驱动到语义驱动的上下文压缩
Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven
AI总结:
本文提出语义一致性上下文压缩方法,通过语义空间动态锚定替代位置驱动,解决位置偏差问题,提升下游任务性能、推理延迟和领域外鲁棒性。
摘要:
大型语言模型(LLMs)在多样化任务中表现出色,但在长上下文场景中面临高计算开销和信息冗余。尽管软提示压缩通过压缩序列到紧凑嵌入来缓解这些成本,现有方法仍受位置偏差限制:依赖可学习标记插入固定位置或按物理标记布局分组,导致性能不稳定和语义碎片化。为克服这一瓶颈,我们提出语义一致性上下文压缩(SeCo),通过语义空间动态锚定替代位置驱动。SeCo不依赖物理标记布局,而是通过选择查询相关标记作为语义中心,并通过一致性加权融合聚合剩余标记。此设计内在保留语义一致性并消除位置偏差。在14个基准测试上对两个基础模型的广泛实验表明,SeCo在下游任务、推理延迟和领域外鲁棒性方面表现优异。代码可在https://anonymous.4open.science/r/seco-EE5E获取。
英文摘要:
Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks. However, their deployment in long-context scenarios faces high computational overhead and information redundancy. While soft prompt compression has emerged as a promising way to mitigate these costs by compressing sequences into compact embeddings, existing paradigms remain fundamentally constrained by position bias: they primarily rely on learnable tokens insertion at fixed positions or group tokens according to their physical token layout, thereby inducing performance instability and semantic fragmentation. To overcome this bottleneck, we propose Semantic Consistency Context Compression (SeCo), a method that shifts context compression from position-driven to semantic-driven. Rather than constraint by physical token layout, SeCo dynamically anchors compression directly in the semantic space by selecting query-relevant tokens as semantic centers and aggregating remaining tokens via consistency-weighted merging. This design inherently preserves semantic consistency while eliminating position bias. Extensive experiments on 14 benchmarks across two backbone models demonstrate that SeCo consistently shows superiority in downstream tasks, inference latency, and out-of-domain robustness. The code is available at https://anonymous.4open.science/r/seco-EE5E.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。