arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.09459cs.CL
收藏

从推理到代理:大型语言模型强化学习中的信用分配

From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models

Independent Researcher(独立研究员)

Chenchen Zhang

AI总结:

本文探讨了大型语言模型强化学习中信用分配问题,总结了47种方法,并提出了可重用的资源,指出从推理到代理的转变使信用分配更加复杂,推动了新的方法发展。

摘要:

强化学习(RL)对于大型语言模型(LLMs)越来越多地依赖稀疏的、结果层面的奖励——但确定长轨迹中哪些动作导致了结果仍然困难。信用分配(CA)问题体现在两个领域:推理RL,其中必须在单个思维链生成(500-30K+ tokens)中的token和步骤之间分配信用;以及代理RL,其中多轮环境交互引入了随机转移、部分可观测性和100+轮的horizons(100K-1M tokens),使episode层面的信用变得越来越不具信息性。我们调查了2024年至2026年初发表的47种CA方法(41种核心方法,6种相邻的使能器),按分配粒度(token、segment、step、turn、multi-agent)和方法(蒙特卡洛、时间差分、模型驱动、博弈论、信息论)进行二维分类。除了调查本身,我们还贡献了三个可重用的资源:(1)一个结构化的、可机读的论文清单,带有分类标签、基线家族和证据等级;(2)一个用于未来CA论文的报告检查清单,经验证后识别系统性方法论缺口;(3)一个基准协议规范,包含任务家族、元数据要求和受控分支任务,附带方法选择决策树。我们的综合分析表明,从推理到代理RL的转变使信用分配景观更加复杂和重塑:推理CA在过程奖励模型和无批评组比较周围成熟,而代理CA推动了真正新的方法——回顾反事实分析、特权不对称批评者和轮级MDP重述——这些方法在推理RL中没有直接先例。

英文摘要:

Reinforcement learning (RL) for large language models (LLMs) increasingly relies on sparse outcome rewards, yet such rewards say little about which token, reasoning step, tool call, memory operation, or agent caused an outcome. This credit assignment (CA) problem spans reasoning RL and becomes sharper in agentic RL, where environment interaction introduces transition non-closure, partial observability, limited replay, heterogeneous actions, weak intermediate verifiability, and agent coupling. We synthesize a unified corpus of 69 papers published from January 2024 through July 31, 2026: 56 core CA methods and 13 adjacent or boundary enablers, selected from 92 deduplicated screening records. We retain the original granularity-by-methodology taxonomy and add a six-diagnostic framework mapping assumption breaks to identification barriers, estimators, and evaluation controls. A source-located full-text audit covers a fixed 42-core-paper subset. Two algorithm researchers independently and blindly cross-coded 252 diagnostic cells, agreeing on 223 (88.5%); per-diagnostic Cohen's kappa ranges from .543 to .909, and principal-family agreement is 42/42 (kappa=1.000). Beyond taxonomy, we establish when restored-state comparisons identify a protocol-specific causal contrast, show that text-only histories can leave even the sign of credit unidentified, and introduce a reusable CA-ID Card linking each claim to its estimand, evidence provenance, and falsification test. An atomic reporting audit describes comparator, budget parity, ablation, overhead, uncertainty, and replay coverage without constructing a cross-paper leaderboard. The companion repository hosts a living catalog and decision aids; a dated release of the frozen audit bundle is planned there separately from the minimal arXiv source.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑