回顾上下文学习用于大语言模型中的时间信用分配
Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models
AI总结：
本文提出利用大语言模型进行回顾上下文学习，以提高强化学习中时间信用分配的样本效率和泛化能力。
摘要：
从自我采样的数据和稀疏环境反馈中学习仍然是训练自我进化代理的基本挑战。时间信用分配通过将稀疏反馈转换为密集的监督信号来缓解这一问题。然而，先前的方法通常依赖于学习任务特定的价值函数进行信用分配，这导致样本效率低下和泛化能力有限。在本工作中，我们提出利用预训练的知识从大语言模型（LLMs）中转换稀疏奖励为密集训练信号（即优势函数）通过回顾上下文学习（RICL）。我们进一步提出一个在线学习框架，RICOL，该框架通过RICL的信用分配结果迭代改进策略。我们实证证明，RICL可以在有限样本下准确估计优势函数，并有效识别环境中的关键状态用于时间信用分配。在四个BabyAI场景上的扩展评估显示，RICOL在传统在线强化学习算法中实现了相当的收敛性能，但具有显著更高的样本效率。我们的发现强调了利用LLMs进行时间信用分配的潜力，为更高效和可泛化的强化学习范式铺平了道路。
英文摘要：
Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on learning task-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pretrained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i.e., the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states in the environment for temporal credit assignment. Extended evaluation on four BabyAI scenarios show that RICOL achieves comparable convergent performance with traditional online RL algorithms with significantly higher sample efficiency. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
