思维简写：通过熵引导的超令牌压缩LLM推理
Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
AI总结：
本文提出通过熵引导的超令牌压缩技术，减少LLM推理过程中的计算开销，同时保留推理结构信息，提升可解释性和效率。
摘要：
大型语言模型的推理过程需要显著的推理时间计算，但推理轨迹的令牌级信息结构仍被忽视。我们发现推理令牌分为两种功能类型：低熵的结构性令牌（重复短语，支撑推理过程）和高熵的有机令牌（问题特定内容，推动解决方案）。这种不对称性促使我们提出一个简单、模型无关的压缩管道：在模型自身的推理轨迹上应用跨词BPE合并，以推导出捕捉频繁结构性模式的超令牌，然后通过监督微调让模型采用它们。在三个模型家族和五个数学推理基准测试中，我们的方法平均缩短推理轨迹8.1%，在任何模型-基准对上都没有显著的精度损失。除了压缩，超令牌还充当可解释的推理移动注释（回溯、验证、策略转换），暴露模型的高层策略。分析结构性类别之间的转换揭示了正确和错误轨迹之间的系统性差异：正确轨迹显示生产性恢复（回溯后跟随策略转换和验证），而错误轨迹则被混淆循环（重复让步和未解决的矛盾）主导。这些诊断信号表明在基于强化学习的推理训练中可用于奖励塑造和提前停止。
英文摘要：
Reasoning in Large Language Models incurs significant inference-time compute, yet the token-level information structure of reasoning traces remains underexplored. We observe that reasoning tokens split into two functional types: low-entropy structural tokens (recurring phrases that scaffold the reasoning process) and higher-entropy organic tokens (problem-specific content that drives toward a solution). This asymmetry motivates a simple, model-agnostic compression pipeline: apply cross-word BPE merges on a model's own reasoning traces to derive \textit{supertokens} that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning. Across three model families and five mathematical reasoning benchmarks, our approach shortens reasoning traces by 8.1% on average; under a TOST equivalence analysis at a +/- 2pp margin, accuracy is equivalent or inconclusive on 13/15 model -- benchmark cells (2 pass equivalence, 11 inconclusive, predominantly AIME at N=30, with non-equivalent degradation on 2/15 cells (DeepSeek-R1-Distill-Llama-70B on MATH-500 and OlympiadBench). Beyond compression, learned supertokens often align with interpretable reasoning moves such as backtracking, verification, and strategy shifts. This enables a compact structural analysis of reasoning traces: correct traces show more recovery and verification patterns, while incorrect traces show more repeated hedging and unresolved counterarguments. We release the full pipeline as open-source code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
