A significant hurdle for current LLMs is the execution of complex, multi-stage tasks. Group Relative Policy Optimization (GRPO) has been emerging as a leading choice, but its reliance on sparse outcome rewards severely limits credit assignment across intermediate steps. Existing remedies such as running full rollouts to assign step-level advantages, calling external LLM judges at each step, or computing intrinsic rewards that require ground-truth answers at every evaluation introduce significant costs or practical constraints. We hypothesize that internal correctness probing over LLM hidden states can be repurposed as a step-level reward signal, potentially addressing all of these limitations at once. However, existing probing research assumes clean inputs, and we first show that this assumption breaks down in multi-step settings: hidden-state probes degrade severely under prefix contamination tracking coherence with the (possibly corrupted) prefix rather than grounded correctness, while attention-based features remain robust to contamination but underperform on clean prefixes. Building on this complementary relationship, we propose the Prefix-Aware Internal Reward (PAIR), a two-stage model with a frozen hidden-state probe estimating belief-consistency and a lightweight attention-based head correcting it toward grounded correctness. Experimental results show that PAIR achieves the highest AUROC on contaminated trajectories while operating at negligible inference cost, enabling dense step-level reward signals for GRPO training without external model calls, ground-truth dependencies, or full-trajectory rollouts.
Rooted Absorbed Prefix Trajectory Balance with Submodular Replay for GFlowNet Training
Courant Institute School of Mathematics, Computing, and Data Science, New York University(纽约大学Courant研究所数学、计算与数据科学学院);Courant Institute School of Mathematics, Computing(纽约大学Courant研究所数学、计算与数据科学学院);Data Science, New York University(纽约大学数据科学学院)
Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所认知与决策智能复杂系统重点实验室);China Merchants Bank(中国 Merchants 银行)
Co-Evolution of Policy and Internal Reward for Language Agents
McGill University(麦吉尔大学);McMaster University(麦克马斯特大学);The University of Hong Kong(香港大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Peking University(北京大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);DeepWisdom(深度智慧);Université de Montréal(蒙特利尔大学);Mila(米拉研究所)
CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery
School of Mathematical Sciences, Beijing University of Posts and Telecommunications(北京邮电大学数学科学学院);School of Mathematics and Statistics, Chongqing University(重庆大学数学与统计学院);School of Science, Beijing Forestry University(北京林业大学理学院)
No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
Zhongguancun Academy(中关村学院);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院)
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,中科院自动化所);School of Artificial Intelligence, UCAS(中国科学技术大学人工智能学院);Fudan University(复旦大学);Independent Researcher(独立研究者)