arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.04550cs.CLcs.AI
收藏

AdmTree: 通过自适应语义树压缩长上下文

AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees

Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Peng Cheng Laboratory(鹏城实验室);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳))

Yangning Li, Shaoshen Chen, Yinghui Li, Yankai Chen, Hai-Tao Zheng, Hui Wang, Wenhao Jiang, Philip S. Yu

AI总结:

AdmTree通过自适应语义树实现高效上下文压缩,保留高语义保真度并减少计算开销。

摘要:

自注意力机制的二次复杂度限制了大型语言模型(LLMs)在处理长上下文时的能力,这种能力对于许多高级应用至关重要。上下文压缩旨在缓解这一计算瓶颈,同时保留关键的语义信息。然而,现有方法往往存在不足:显式方法可能牺牲局部细节,而隐式方法可能遭受位置偏见、信息退化或无法捕捉长距离语义依赖。我们提出了AdmTree,一种新的自适应、分层上下文压缩框架,其核心目标是保持高语义保真度的同时维持效率。AdmTree根据信息密度动态分割输入,利用概要标记将可变长度段作为语义二叉树的叶子节点进行总结。这种结构,结合轻量级聚合机制和冻结的骨干LLM(从而最小化新的可训练参数),使上下文的分层抽象变得高效。通过在保留细粒度细节的同时保持全局语义一致性,缓解位置偏见,并动态适应内容,AdmTree稳健地保留了长上下文的语义信息。

英文摘要:

The quadratic complexity of self-attention constrains Large Language Models (LLMs) in processing long contexts, a capability essential for many advanced applications. Context compression aims to alleviate this computational bottleneck while retaining critical semantic information. However, existing approaches often fall short: explicit methods may compromise local detail, whereas implicit methods can suffer from positional biases, information degradation, or an inability to capture long-range semantic dependencies. We propose AdmTree, a novel framework for adaptive, hierarchical context compression with a central focus on preserving high semantic fidelity while maintaining efficiency. AdmTree dynamically segments input based on information density, utilizing gist tokens to summarize variable-length segments as the leaves of a semantic binary tree. This structure, together with a lightweight aggregation mechanism and a frozen backbone LLM (thereby minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By preserving fine-grained details alongside global semantic coherence, mitigating positional bias, and dynamically adapting to content, AdmTree robustly retains the semantic information of long contexts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑