评估大型语言模型在强化学习中行动评估的零样本能力
Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RL
AI总结：
本研究提出CALM框架，利用大型语言模型在零样本条件下通过奖励塑形和选项发现自动化信用分配，初步实验表明其能有效增强稀疏延迟奖励下的强化学习。
摘要：
时间信用分配问题是强化学习（RL）中的一个核心挑战，涉及将轨迹中每个行动对实现目标的能力的适当影响归因。然而，当反馈延迟且稀疏时，学习信号较差，行动评估变得更加困难。典型解决方案，如奖励塑形和选项，需要大量的领域知识和人工干预，限制了其可扩展性和适用性。在这项工作中，我们为基于语言模型的信用分配（CALM）奠定了基础，这是一种新颖的方法，利用大型语言模型（LLMs）通过奖励塑形和选项发现来自动化信用分配。CALM使用LLMs将任务分解为基本子目标，并评估这些子目标在状态-行动转换中的达成情况。每当一个选项终止时，一个子目标达成，CALM提供辅助奖励。当任务奖励稀疏且延迟时，这种额外的奖励信号可以增强学习过程，无需人工设计的奖励。我们使用来自MiniHack的人工标注演示数据集对CALM进行了初步评估，表明LLMs在零样本设置中无需示例或LLM微调即可有效分配信用。我们的初步结果表明，LLMs的知识是RL中信用分配的有前景的先验，促进了人类知识向价值函数的转移。
英文摘要：
The temporal credit assignment problem is a central challenge in Reinforcement Learning (RL), concerned with attributing the appropriate influence to each actions in a trajectory for their ability to achieve a goal. However, when feedback is delayed and sparse, the learning signal is poor, and action evaluation becomes harder. Canonical solutions, such as reward shaping and options, require extensive domain knowledge and manual intervention, limiting their scalability and applicability. In this work, we lay the foundations for Credit Assignment with Language Models (CALM), a novel approach that leverages Large Language Models (LLMs) to automate credit assignment via reward shaping and options discovery. CALM uses LLMs to decompose a task into elementary subgoals and assess the achievement of these subgoals in state-action transitions. Every time an option terminates, a subgoal is achieved, and CALM provides an auxiliary reward. This additional reward signal can enhance the learning process when the task reward is sparse and delayed without the need for human-designed rewards. We provide a preliminary evaluation of CALM using a dataset of human-annotated demonstrations from MiniHack, suggesting that LLMs can be effective in assigning credit in zero-shot settings, without examples or LLM fine-tuning. Our preliminary results indicate that the knowledge of LLMs is a promising prior for credit assignment in RL, facilitating the transfer of human knowledge into value functions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
