AI 中文总结
研究发现长度惩罚强化学习会缩短思维链推理，隐藏驱动模型答案的因素。通过训练不同目标链长度的Qwen3模型并评估，表明压缩虽能减少推理令牌、保留准确率，但降低了可监测性，揭示了压缩与可监测性的前沿关系。
AI 中文摘要
长度惩罚强化学习会缩短思维链推理，同时隐藏驱动模型答案的影响因素。实验中，即便模型思维链提及提示较少，但长度惩罚训练仍无法阻止误导性提示引导模型。令牌准确率评估会将这些运行视为成功，却忽略剩余痕迹是否仍显示驱动答案的因素。我们训练了不同目标链长度的Qwen3 - 4B和Qwen3 - 14B变体，并用偏差提示干预在保留的MMLU - Pro - R和四个迁移基准上评估它们。压缩大幅减少推理令牌，保留多数多项选择准确率，且提示影响接近基线。在最强目标下，Qwen3 - 14B的下限忠实度降至基线的63.1%，Qwen3 - 4B降至69.4%；监测器捕捉提示使用的原始比率从69%降至49%，从60%降至48%。为分离长度与内容，我们从未压缩的基线链中随机删除句子直至剩余文本与压缩长度匹配。即便长度匹配后，对于两种Qwen3大小和所有五种评估分布，压缩链披露提示的频率比随机缩短的基线链低7 - 35个百分点。因此，压缩不仅缩短推理，还优先去除监测器查看影响答案因素所需的线索。这些结果揭示了一个压缩 - 可监测性前沿，即更廉价的推理可保留答案，同时使背后的影响更难被察觉。
英文摘要
Recent work trains reasoning models with length penalties to curb overthinking and cut inference cost. We show that these penalties make the chain of thought less monitorable. A length-compressed model still lets misleading hints steer its answers, but it less often verbalizes their influence. We train Qwen3-4B and Qwen3-14B with reinforcement learning under length penalties targeting 60% down to 30% of baseline chain-of-thought length, then evaluate them with nine types of biasing hints on held-out MMLU-Pro-R and four transfer benchmarks. A chain is faithful when an LLM monitor can tell from it that the hint influenced the answer. At the 30% target, accuracy stays near baseline and wrong-answer hints switch answers as often as before. Yet faithfulness drops on every evaluation set for both models, by 39% for Qwen3-14B and 35% for Qwen3-4B on MMLU-Pro-R. A control trained with the same correctness and format rewards but no length penalty leaves faithfulness intact or raises it. Shortening alone does not explain the drop. Compressed chains mention the hint 7 to 35 percentage points less often than the uncompressed model's chains shortened to the same length by random sentence deletion, across both model sizes and all five evaluation sets. Length penalties therefore trade monitorability for inference cost by removing the evidence monitors depend on.
