超越密集状态：将稀疏转码器提升为活跃运算符以进行潜在推理
Beyond Dense States: Sparse Transcoders as Causally Testable Operators for LLM Latent Reasoning
AI总结：
LSTR通过将稀疏转码器提升为活跃运算符，实现更可控和可解释的潜在推理。
摘要：
潜在推理将思维链（CoT）压缩为连续隐状态，但现有方法依赖于密集的隐状态转移，仍然难以解释和控制。同时，稀疏表示模型揭示了人类可解释的语义特征，但仍然主要局限于事后分析。我们通过提出LSTR（潜在稀疏转码器推理），一种潜在推理框架，将功能稀疏转码器提升为活跃推理运算符，通过稀疏语义转移进行多步计算。其核心是采用具有残差跳过架构的潜在过渡转码器（LTT），将线性流形运输与稀疏语义更新解耦，通过显式稀疏约束实现可控的语义解析。大量实验表明，LSTR在保持推理准确性和压缩效率的同时，显著优于密集隐状态基线的可解释性。因果干预和轨迹分析进一步表明，这些稀疏特征在推理过程中既是可解释的又是因果有效的运算符。
英文摘要：
Latent reasoning reduces the token-generation cost of chain-of-thought reasoning by replacing explicit intermediate tokens with continuous latent transitions. However, existing latent reasoning methods usually rely on dense and entangled transitions, making their reasoning trajectories difficult to inspect or intervene on. We introduce LSTR (Latent Sparse Transcoder Reasoning), a framework that turns sparse transcoders from post-hoc diagnostic tools into in-loop, intervenable transition components for latent reasoning. At each latent step, a Latent Transition Transcoder (LTT) combines a linear skip path with a Top-k sparse innovation path, exposing a small set of active sparse features. Under matched compression settings, LSTR offers a mechanistically inspectable alternative to dense latent reasoning. On GSM8K-Aug, ablating only a few top-active sparse features reduces accuracy by up to 16.5%, whereas analogous interventions have much smaller effects in dense latent baselines. These results indicate that the active sparse features are causally involved in the latent transition process, rather than merely post-hoc descriptors. Additional experiments on mathematical benchmarks and StrategyQA suggest that sparse latent transitions can preserve the compression benefits of latent reasoning while making the resulting trajectories more inspectable and intervenable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
