揭秘大型推理模型中基于熵的思维链压缩选择方法

中文摘要 AI 辅助

基于熵的剪枝被提出作为一种有效压缩思维链（CoT）推理的方法，其精度损失可忽略不计。我们在多种模型和推理任务中测试了低熵与高熵CoT步骤选择方法的鲁棒性，结果显示在所有评估设置中，熵方法相比随机剪枝无任何优势。随后我们从句子层面转向令牌层面，发现保留低熵令牌仅在数学基准上有效，原因是数学问题中的数字令牌本身具有低熵特性，且还承载语义内容。最后我们证明，用原始激活值修补少量CoT令牌的子集可恢复近乎完美的全轨迹性能，这提供了因果证据：任务信息并非集中在启发式可识别的小部分CoT令牌中，而是分布在整个推理链上。

英文摘要

Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from sentences to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.

发表机构

• University of Trieste（的里雅斯特大学）
• University of Milano-Bicocca（米兰-比科卡大学）
• University of Groningen（格罗宁根大学）
• Khoury College of Computer Sciences, Northeastern University（东北大学科里计算机学院）

机构由 AI 辅助整理，请以论文原文为准。