LEPO：面向大语言模型的潜在推理策略优化
LEPO: Latent Reasoning Policy Optimization for Large Language Models
AI总结：
LEPO通过引入Gumbel-Softmax在大语言模型中实现可控的随机性，提升其探索能力与强化学习兼容性，通过直接在连续潜在表示上应用强化学习，显著优于现有方法。
摘要：
近年来，潜在推理被引入大语言模型（LLMs）以利用连续空间中的丰富信息。然而，缺乏随机采样时，这些方法不可避免地退化为确定性推理，无法发现多样的推理路径。为弥合这一差距，我们通过Gumbel-Softmax在潜在推理中注入可控的随机性，恢复LLMs的探索能力并增强其与强化学习（RL）的兼容性。在此基础上，我们提出LEPO，一种将强化学习直接应用于连续潜在表示的新框架。具体而言，在回放阶段，LEPO保持随机性以实现多样化的轨迹采样；在优化阶段，LEPO为潜在表示和离散令牌构建统一的梯度估计。大量实验表明，LEPO在离散和潜在推理方面显著优于现有RL方法。
英文摘要：
Recently, latent reasoning has been introduced into large language models (LLMs) to leverage rich information within a continuous space. However, without stochastic sampling, these methods inevitably collapse to deterministic inference, failing to discover diverse reasoning paths. To bridge the gap, we inject controllable stochasticity into latent reasoning via Gumbel-Softmax, restoring LLMs' exploratory capacity and enhancing their compatibility with Reinforcement Learning (RL). Building on this, we propose \textbf{\underline{L}}atent R\textbf{\underline{e}}asoning \textbf{\underline{P}}olicy \textbf{\underline{O}}ptimization~(\textbf{LEPO}), a novel framework that applies RL directly to continuous latent representations. Specifically, in rollout stage, LEPO maintains stochasticity to enable diverse trajectory sampling, while in optimization stage, LEPO constructs a unified gradient estimation for both latent representations and discrete tokens. Extensive experiments show that LEPO significantly outperforms existing RL methods for discrete and latent reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
