From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
Independent Researcher(独立研究员)
Chenchen Zhang
备注:
50 pages, 4 figures. v3: expanded to a unified 69-paper corpus through July 31, 2026; adds restored-state identification results, blind coding of a 42-paper full-text subset, a source-located reporting audit, the CA-ID Card, and replay-fidelity analysis
Reinforcement learning (RL) for large language models (LLMs) increasingly relies on sparse outcome rewards, yet such rewards say little about which token, reasoning step, tool call, memory operation, or agent caused an outcome. This credit assignment (CA) problem spans reasoning RL and becomes sharper in agentic RL, where environment interaction introduces transition non-closure, partial observability, limited replay, heterogeneous actions, weak intermediate verifiability, and agent coupling.
We synthesize a unified corpus of 69 papers published from January 2024 through July 31, 2026: 56 core CA methods and 13 adjacent or boundary enablers, selected from 92 deduplicated screening records. We retain the original granularity-by-methodology taxonomy and add a six-diagnostic framework mapping assumption breaks to identification barriers, estimators, and evaluation controls. A source-located full-text audit covers a fixed 42-core-paper subset. Two algorithm researchers independently and blindly cross-coded 252 diagnostic cells, agreeing on 223 (88.5%); per-diagnostic Cohen's kappa ranges from .543 to .909, and principal-family agreement is 42/42 (kappa=1.000).
Beyond taxonomy, we establish when restored-state comparisons identify a protocol-specific causal contrast, show that text-only histories can leave even the sign of credit unidentified, and introduce a reusable CA-ID Card linking each claim to its estimand, evidence provenance, and falsification test. An atomic reporting audit describes comparator, budget parity, ablation, overhead, uncertainty, and replay coverage without constructing a cross-paper leaderboard. The companion repository hosts a living catalog and decision aids; a dated release of the frozen audit bundle is planned there separately from the minimal arXiv source.
My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning
Alibaba Group(阿里巴巴集团);Kyoto University(京都大学);NII LLMC(国立情报学研究所LLMC);Peking University(北京大学);University of California, Los Angeles(加州大学洛杉矶分校);Arizona State University(亚利桑那州立大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tsinghua University(清华大学);University of the Chinese Academy of Sciences(中国科学院大学)
CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic
Institute of Automation, CAS(中国科学院自动化研究所);School of Advanced Interdisciplinary Sciences, UCAS(清华大学交叉信息学院);School of Artificial Intelligence, UCAS(清华大学人工智能学院)
FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection
Zhejiang University(浙江大学);Kuaishou Technology(快手科技);University of the Chinese Academy of Sciences(中国科学院大学);Zhejiang University of Science and Technology(浙江科技学院)
Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning
Fudan University(复旦大学);XingYun lab, HUJING Digital Media & Entertainment Group(星云实验室,HUJING数字媒体与娱乐集团);University of Science and Technology Beijing(北京科技大学);Chinese Academy of Sciences(中国科学院);Beijing University of Posts and Telecommunications(北京邮电大学)
Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL
Institute of Computer Science and Technology, Soochow University, China(苏州大学计算机科学与技术学院);Ant Digital Technologies, Ant Group(蚂蚁集团数字技术部);School of Management, University of Science and Technology of China(中国科学技术大学管理学院)
Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents
University of California, Irvine(加利福尼亚大学欧文分校);Georgia Institute of Technology(佐治亚理工学院);Dartmouth College(达特茅斯学院);Northeastern University(东北大学);University of Notre Dame(圣母大学)
APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation
University of Chinese Academy of Sciences(中国科学院大学);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS部);Wenge Technology Co., Ltd(Wenger科技有限公司)
Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Alibaba Group(阿里巴巴集团);Peking University(北京大学)
ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learning
University of Chinese Academy of Sciences(中国科学院大学);Institute of Neuroscience, Chinese Academy of Sciences(中国科学院神经科学研究所);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Nanyang Technological University(南洋理工大学)