思维链压缩：理论分析
Chain Of Thought Compression: A Theoretical Analysis
AI总结：
本文通过引入Order-r Interaction理论，证明了隐式思维链压缩中高阶逻辑依赖的学习信号指数衰减问题，并提出ALiCoT框架通过对齐潜在令牌分布与中间推理状态来克服信号衰减，实现54.4倍加速且性能与显式CoT相当。
摘要：
思维链（CoT）通过中间步骤解锁了大语言模型（LLMs）的高级推理能力，但由于生成额外令牌而带来了高昂的计算成本。最近的研究经验表明，将推理步骤压缩到潜在状态中，即隐式CoT压缩，提供了一种令牌高效的替代方案。然而，CoT压缩背后的机制仍不清楚。在本文中，我们首次对学习内化中间推理步骤的难度进行了理论分析。通过引入Order-r Interaction，我们证明了高阶逻辑依赖的学习信号指数衰减以解决不可约问题，其中跳过中间步骤不可避免地导致高阶交互障碍。为了经验验证这一点，我们引入了NatBool-DAG，这是一个具有挑战性的基准测试，旨在强制执行不可约逻辑推理并消除语义捷径。在我们的理论发现指导下，我们提出了ALiCoT（对齐隐式CoT），一种新颖的框架，通过对齐潜在令牌分布与中间推理状态来克服信号衰减。实验结果表明，ALiCoT成功解锁了高效推理：它实现了54.4倍加速，同时保持与显式CoT相当的性能。
英文摘要：
Chain-of-Thought (CoT) has unlocked advanced reasoning abilities of Large Language Models (LLMs) with intermediate steps, yet incurs prohibitive computational costs due to generation of extra tokens. Recent studies empirically show that compressing reasoning steps into latent states, or implicit CoT compression, offers a token-efficient alternative. However, the mechanism behind CoT compression remains unclear. In this paper, we provide the first theoretical analysis of the difficulty of learning to internalize intermediate reasoning steps. By introducing Order-r Interaction, we prove that the learning signal for high-order logical dependencies exponentially decays to solve irreducible problem, where skipping intermediate steps inevitably leads to high-order interaction barriers. To empirically validate this, we introduce NatBool-DAG, a challenging benchmark designed to enforce irreducible logical reasoning and eliminate semantic shortcuts. Guided by our theoretical findings, we propose ALiCoT (Aligned Implicit CoT), a novel framework that overcomes the signal decay by aligning latent token distributions with intermediate reasoning states. Experimental results demonstrate that ALiCoT successfully unlocks efficient reasoning: it achieves a 54.4x speedup while maintaining performance comparable to explicit CoT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
