潜在思维优化：你的潜在推理语言模型秘密在潜在思维中编码了奖励信号
Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts
AI总结：
本文提出LTO方法，通过潜在奖励模型优化LLMs的潜在推理过程，提升推理效率与准确性。
摘要：
大型语言模型（LLMs）通过生成自然语言的推理链来解决复杂问题，但这种口头推理过程计算成本高且容易过度思考。最近的一项工作提出了一种潜在推理架构Huginn-3.5B，将中间推理步骤表示为一组潜在表示。然而，潜在思维缺乏可解释性且难以监督，引发了对模型潜在推理过程正确性和可靠性的担忧。本文系统研究了Huginn-3.5B在潜在空间中的推理方式以及外部监督信号如何改进其潜在推理过程。我们发现，导致正确或错误答案的潜在思维表现出高度可区分的模式，且潜在分类器能够从潜在思维中可靠地预测答案的正确性。基于这些见解，我们提出了潜在思维优化（LTO），一种概率算法，利用潜在分类器作为潜在奖励模型（LRM）来优化潜在推理过程。在多样化的推理任务上进行的广泛实验表明，LRM在检测错误的潜在推理模式方面效果显著，LTO能显著改进潜在推理过程。此外，我们展示了LRM能够跨不同领域泛化，LTO可以无缝应用于通用LLMs以改进其推理过程。与口头推理相比，我们的方法表明，奖励建模和在测试时通过监督扩展推理过程可以直接在潜在空间中进行，突显了其作为通用、高效且领域无关的改进LLMs推理过程的方法的潜力。
英文摘要：
Large Language Models (LLMs) excel at problem solving by generating chain of thoughts in natural language, but such verbal thinking is computationally costly and prone to overthinking. A recent work instead proposes a latent thinking architecture, Huginn-3.5B, which represents intermediate reasoning steps as a sequence of latent representations. However, latent thoughts lack interpretability and are difficult to supervise, raising concerns about the correctness and reliability of the model's latent thinking processes. In this paper, we provide a systematic study of how Huginn-3.5B thinks in the latent space and how external supervision signals can improve its latent thinking processes. We show that latent thoughts leading to correct versus incorrect answers exhibit highly distinguishable patterns, and that a latent classifier can reliably predict answer correctness directly from latent thoughts. Leveraging these insights, we propose Latent Thinking Optimization (LTO), a probabilistic algorithm that employs the latent classifier as a Latent Reward Model (LRM) to optimize the latent thinking processes. Extensive experiments across diverse reasoning tasks demonstrate that LRM is highly effective in detecting incorrect latent thinking patterns, and LTO can significantly improve the latent thinking processes. Furthermore, we show that LRM can generalize across diverse domains, and LTO can be seamlessly applied to general LLMs to improve their thinking processes. In contrast to verbal thinking, our method demonstrates that reward modeling and scaling test-time thinking with supervision can be performed directly in the latent space, highlighting its potential as a general, efficient, and domain-agnostic approach to improving the thinking processes of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
