Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China（中国人民大学北京校区人工智能学院）；MiLM Plus, Xiaomi Inc., Beijing, China（小米公司北京MiLM Plus团队）；University of Modena and Reggio Emilia（莫德纳和雷吉奥艾米莉亚大学）；University of Pisa, Italy（比萨大学）
Large Reasoning Models (LRMs) have recently achieved strong mathematical and code reasoning performance through Reinforcement Learning (RL) post-training. However, we show that modern reasoning post-training induces an unintended exploration collapse: temperature-based sampling no longer increases pass@$n$ accuracy. Empirically, the final-layer posterior of post-trained LRMs exhibit sharply reduced entropy, while the entropy of intermediate layers remains relatively high. Motivated by this entropy asymmetry, we propose Latent Exploration Decoding (LED), a depth-conditioned decoding strategy. LED aggregates intermediate posteriors via cumulative sum and selects depth configurations with maximal entropy as exploration candidates. Without additional training or parameters, LED consistently improves pass@1 and pass@16 accuracy by 0.61 and 1.03 percentage points across multiple reasoning benchmarks and models. Furthermore, integrating LED into reinforcement learning, e.g., using GRPO as the rollout strategy, yields faster reward improvement and higher final performance, due to the efficient exploration capability of LED. Project page: https://github.com/AlbertTan404/LED.
Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research, New York（微软研究院（纽约））；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research, Redmond（微软研究院（雷德蒙德））
We study the exploration problem in episodic MDPs with rich observations generated from a small number of latent states. Under certain identifiability assumptions, we demonstrate how to estimate a mapping from the observations to latent states inductively through a sequence of regression and clustering steps -- where previously decoded latent states provide labels for later regression problems -- and use it to construct good exploration policies. We provide finite-sample guarantees on the quality of the learned state decoding function and exploration policies, and complement our theory with an…
Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
Department of Computer Science, University of Chicago（芝加哥大学计算机科学系）；Department of Statistics, University of Chicago（芝加哥大学统计系）；Toyota Technological Insitute at Chicago（芝加哥丰田技术研究所）；Data Science Institute, University of Chicago（芝加哥大学数据科学研究所）；Meta AI
New York University（纽约大学）；Chalmers University of Technology（查尔姆斯理工大学）；University of Parma（帕尔马大学）；Duke University（杜克大学）
In this paper, we use reinforcement learning to find effective decoding strategies for binary linear codes. We start by reviewing several iterative decoding algorithms that involve a decision-making process at each step, including bit-flipping (BF) decoding, residual belief propagation, and anchor decoding. We then illustrate how such algorithms can be mapped to Markov decision processes allowing for data-driven learning of optimal decision strategies, rather than basing decisions on heuristics or intuition. As a case study, we consider BF decoding for both the binary symmetric and additive wh…
Seoul National University（首尔大学）；Neural Processing Research Center（神经处理研究中心）；UC Berkeley（加州大学伯克利分校）
Reinforcement learning algorithms struggle when the reward signal is very sparse. In these cases, naive random exploration methods essentially rely on a random walk to stumble onto a rewarding state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or discriminative modeling of novelty. We propose EMI, which is an exploration method that constructs embedding representation of states and actions that does not rely on generative decoding of the full observation but extracts predictive signals that can be used to guide exploratio…
$ϕ$-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time Exploration and Exploitation
Shanghai AI Lab（上海人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）；National University of Singapore（新加坡国立大学）
Open-ended text generation with autoregressive language models (LMs) is one of the core tasks in natural language processing. However, maximization-based decoding methods (e.g., greedy/beam search) often lead to the degeneration problem, i.e., the generated text is unnatural and contains undesirable repetitions. Existing solutions to this problem either introduce randomness prone to incoherence or require a look-ahead mechanism that demands extra computational overhead. In this study, we formulate open-ended text generation from a new perspective, i.e., we view it as an exploration process wit…
