通过内在显著性剪枝压缩链式推理中的冗余
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning
AI总结：
本文提出CRISP框架，通过利用模型内在显著性信号压缩链式推理，减少冗余并保持逻辑一致性，实验表明在不牺牲准确性的情况下实现50-60%的token减少。
摘要：
长链式推理（CoT）对最近推理模型的成功至关重要，但存在高计算开销和延迟。先前工作尝试通过外部压缩器压缩CoT，但往往无法与模型内部推理动态对齐，导致关键逻辑步骤的丢失。本文提出了通过内在显著性剪枝压缩链式推理的CRISP框架，分析发现推理终止标记[object Object]作为信息锚点，其注意力模式有效区分了必要推理和冗余。基于此发现，我们设计了一种利用这些内在注意力信号指导原子压缩操作的策略。与粗粒度剪枝策略相比，CRISP战略性地提炼推理链以最大化信息密度并保持逻辑一致性。在各种基础模型和数学数据集上的实验证明，CRISP在不牺牲准确性的情况下实现了50-60%的token减少，有效缓解了长上下文推理的效率瓶颈。我们开源了我们的实现以促进高效推理的进一步研究。
英文摘要：
Long Chain-of-Thought (CoT) reasoning is pivotal for the success of recent reasoning models but suffers from high computational overhead and latency. While prior works attempt to compress CoT via external compressor, they often fail to align with the model's internal reasoning dynamics, resulting in the loss of critical logical steps. This paper presents \textbf{C}ompressing \textbf{R}edundancy in Chain-of-Thought via \textbf{I}ntrinsic \textbf{S}aliency \textbf{P}runing (\textbf{CRISP}), a framework that compresses CoT by exploiting the model's intrinsic saliency. Our analysis reveals a distinct phenomenon: the reasoning termination token \texttt{[object Object]} acts as an information anchor, where its attention pattern effectively demarcates essential reasoning from redundancy. Based on this finding, we design a policy that utilizes these intrinsic attention signals to guide atomic compression operations. In contrast to coarse-grained pruning strategies, CRISP strategically distills the reasoning chain to maximize information density while preserving logical coherence. Empirical results across various backbone models and mathematical datasets demonstrate that CRISP achieves a 50-60% reduction in token count without compromising accuracy, effectively mitigating the efficiency bottleneck of long-context reasoning. We open-source our implementation to facilitate further research in efficient reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
