发表机构
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究通过实验发现直接上下文强化学习（ICRL）中奖励影响极小，轨迹推动性能提升的关键并非语义内容，直接ICRL更应视为上下文学习（ICL）的特例，为智能体记忆设计提供了新方向。
AI 中文摘要
大语言模型智能体越来越多地通过在上下文积累经验而非更新参数来提升推理能力，这一过程常被称为上下文强化学习（ICRL）。然而，上下文学习（ICL）是否真能发挥强化学习（RL）的作用尚未得到验证。我们以最简单的直接ICRL形式研究该问题，其中模型直接以原始轨迹-奖励对为条件，探究奖励是否作为学习信号。通过在六个模型上对四个基准开展受控实验，我们发现奖励确实被读取，但其影响很小：翻转、随机化或移除奖励后，提升曲线几乎保持不变，即便在明确指示模型探索、利用或推理奖励的元提示下亦是如此。轨迹推动性能提升，但并非通过其语义内容：打乱或损坏的轨迹与真实轨迹效果相当。这些模式与ICL中已知的情况高度相似，表明直接ICRL更适合被理解为ICL的特例，而非推理时的RL。这一重新界定对智能体记忆设计具有启示：输入分布和演示等ICL因素可能比奖励塑造和探索等RL因素更重要。
英文摘要
LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters. This process is often described as in-context reinforcement learning (ICRL). Whether in-context learning (ICL) can actually play the role of RL, however, has not been tested. We study this question in its simplest form, direct ICRL, where the model conditions directly on raw trajectory-reward pairs, and ask whether the reward acts as a learning signal. Through controlled experiments on four benchmarks across six models, we find that the reward is read, but its effect is small: flipping, randomizing, or removing the reward leaves the improvement curve almost unchanged, and this holds even under meta-prompts that explicitly instruct the model to explore, exploit, or reason over rewards. Trajectories drive improvement, but not through their semantic content: shuffled or corrupted trajectories work as well as real ones. These patterns closely mirror those known in ICL, suggesting that direct ICRL is better understood as a special case of ICL than as inference-time RL. This reframing has implications for agent memory design: ICL factors such as input distribution and demonstrations may matter more than RL elements such as reward shaping and exploration.
CommentsNeurIPS 2026 Spotlight (Negative Results Track)
