Latent Reward：情景强化学习中由 LLM 赋能的信用分配
Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning
AI总结：
针对情景强化学习中稀疏奖励与信用归因难题，本文提出 LLM 赋能的符号决策框架 LaRe，通过可执行代码生成多维 Latent Reward 并自验证，提升单智能体与多智能体信用分配效果。
摘要：
强化学习（RL）在现实应用中经常遇到延迟且稀疏的反馈，即使仅有情景奖励也是如此。以往方法在用于信用分配的奖励再分配方面取得了一定进展，但仍面临诸多挑战，包括由冗余导致的训练困难，以及因忽视任务绩效评估的多面性而产生的模糊归因。令人期待的是，大语言模型（LLM）包含丰富的决策知识，并为奖励再分配提供了一种可行工具。即便如此，由于语言知识与符号形式要求之间的错位，以及推理过程中固有的随机性和幻觉，在这种情形下部署 LLM 并非易事。为解决这些问题，我们提出 LaRe，这是一种新颖的、由 LLM 赋能且基于符号的决策框架，用于改进信用分配。LaRe 的关键是 Latent Reward 概念，它充当多维绩效评估，能够从不同视角实现更具可解释性的目标达成，并促进更有效的奖励再分配。我们验证了由 LLM 生成的语义代码可以连接语言知识与符号化 latent rewards，因为这类代码可针对符号对象执行。同时，我们设计了 latent reward 自验证，以提高 LLM 推理的稳定性和可靠性。理论上，消除 latent reward 中与奖励无关的冗余，可通过更准确的奖励估计使 RL 性能受益。大量实验结果表明，LaRe：（i）在时间信用分配上优于 SOTA 方法；（ii）擅长在多个智能体之间分配贡献；（iii）在某些任务上优于使用真实奖励训练的策略。
英文摘要：
Reinforcement learning (RL) often encounters delayed and sparse feedback in real-world applications, even with only episodic rewards. Previous approaches have made some progress in reward redistribution for credit assignment but still face challenges, including training difficulties due to redundancy and ambiguous attributions stemming from overlooking the multifaceted nature of mission performance evaluation. Hopefully, Large Language Model (LLM) encompasses fruitful decision-making knowledge and provides a plausible tool for reward redistribution. Even so, deploying LLM in this case is non-trivial due to the misalignment between linguistic knowledge and the symbolic form requirement, together with inherent randomness and hallucinations in inference. To tackle these issues, we introduce LaRe, a novel LLM-empowered symbolic-based decision-making framework, to improve credit assignment. Key to LaRe is the concept of the Latent Reward, which works as a multi-dimensional performance evaluation, enabling more interpretable goal attainment from various perspectives and facilitating more effective reward redistribution. We examine that semantically generated code from LLM can bridge linguistic knowledge and symbolic latent rewards, as it is executable for symbolic objects. Meanwhile, we design latent reward self-verification to increase the stability and reliability of LLM inference. Theoretically, reward-irrelevant redundancy elimination in the latent reward benefits RL performance from more accurate reward estimation. Extensive experimental results witness that LaRe (i) achieves superior temporal credit assignment to SOTA methods, (ii) excels in allocating contributions among multiple agents, and (iii) outperforms policies trained with ground truth rewards for certain tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
