Reinforcement learning has emerged as an effective paradigm for training large language models to interleave reasoning with search engine calls. However, existing approaches face a fundamental credit assignment problem: methods like Search-R1 assign a single outcome reward to the entire multi-step trajectory, providing no signal about which reasoning or retrieval decisions were responsible for success or failure. Process-reward methods such as StepSearch introduce step-level supervision but still sample complete trajectories independently, so advantage estimates at any given step are contaminated by the randomness of all other steps. We propose SLATE (Step-Level Advantage estimation for Truncated Exploration), which addresses both problems through two complementary ideas. First, truncated step-level sampling generates k continuations from a shared prefix, isolating all variation to a single decision point. We prove this reduces the variance of advantage estimates by up to a factor of T compared to full-trajectory sampling for T-step trajectories, the first formal variance guarantee for step-level RL in retrieval-augmented reasoning. Second, dense, decomposed process rewards separately evaluate reasoning quality, query quality, and answer correctness on a ternary scale via an LLM judge, providing richer supervision than binary outcome signals or heuristic step-level scores. Experiments on seven QA benchmarks show that SLATE consistently outperforms both sparse-reward and process-reward baselines, achieving a 7.0% relative improvement over Search-R1 on the 7B model and 30.7% on the 3B model. Gains are largest on challenging multi-hop tasks, and ablations confirm that truncated sampling and dense rewards provide complementary benefits.
Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?
IIIS, Tsinghua University(清华大学信息学院);AI Lab, Princeton University(普林斯顿大学人工智能实验室);Department of Computer Science & Engineering, University of Michigan(密歇根大学计算机科学与工程系)
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
Kuaishou Technology Co., Ltd.(快手科技有限公司);Institute for Clarity in Documentation(文献清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康分所);Rajiv Gandhi University(拉吉夫甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算高精尖创新中心);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Beijing Academy of Blockchain and Edge Computing(北京区块链与边缘计算研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学深圳研究院);Institute of Artificial Intelligence (TeleAI), China Telecom Corp Ltd(中国电信人工智能研究院);College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China(浙江大学集成电路学院);Zhongguancun Academy, Beijing, China(中关村学院)
AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
The University of Texas at Dallas(德克萨斯大学达拉斯分校);Adobe Inc.(Adobe公司);Department of Computer Science, University of California, Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation
Laboratory of Speech and Intelligent Information Processing, Institute of Acoustics, Chinese Academy of Sciences(语音与智能信息处理实验室,声学研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);China Southern Power Grid Artificial Intelligence Technology Co., Ltd.(中国南方电网人工智能技术有限公司)
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
The Graduate University for Advanced Studies (SOKENDAI)(高级研究大学(SOKENDAI));National Institute of Informatics(国家信息研究所);The Asahi Shimbun Company(朝日新闻公司);The University of Tokyo(东京大学)
ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Chinese Academy of Sciences(中国科学院);Tsinghua University(清华大学)
APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation
University of Chinese Academy of Sciences(中国科学院大学);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS部);Wenge Technology Co., Ltd(Wenger科技有限公司)
ARAML: A Stable Adversarial Training Framework for Text Generation
Institute for Artificial Intelligence(人工智能研究院);State Key Lab of Intelligent Technology and Systems(智能技术与系统国家重点实验室);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
Language-Coupled Reinforcement Learning for Multilingual Retrieval-Augmented Generation
School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学);University of Montreal(蒙特利尔大学);Tsinghua University(清华大学);University of Notre Dame(Notre Dame 大学)