IQL-TD-MPC：面向分层模型预测控制的隐式Q学习
IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control
AI总结：
针对离线场景下长序列稀疏奖励任务中基于模型的强化学习长期规划能力不足的问题，提出融合隐式Q学习的离线模型预测控制算法IQL-TD-MPC，并将其作为分层架构的管理者增强状态表示，显著提升了D4RL基准任务的性能。
摘要：
基于模型的强化学习（RL）凭借其样本效率展现出巨大潜力，但在长序列稀疏奖励任务中仍存在困难，尤其是在智能体从固定数据集学习的离线场景下。我们假设基于模型的RL智能体在这类环境中表现不佳是由于缺乏长期规划能力，而在环境的时间抽象模型中进行规划能够缓解这一问题。本文提出两项核心贡献：1）提出一种离线基于模型的RL算法IQL-TD-MPC，将隐式Q学习（IQL）融入当前最优的模型预测控制时序差分学习（TD-MPC）进行扩展；2）提出将IQL-TD-MPC作为分层架构中的管理者，搭配任意现成的离线RL算法作为工作者。具体而言，我们预训练一个时间抽象的IQL-TD-MPC管理者，通过规划预测大致对应子目标的“意图嵌入”。实验结果表明，用IQL-TD-MPC管理者生成的意图嵌入增强状态表示，能显著提升现成离线RL智能体在部分极具挑战性的D4RL基准任务上的表现。例如，离线RL算法AWAC、TD3-BC、DT和CQL在中等和大型antmaze任务上的归一化评估得分为零或接近零，而我们的改进方案平均得分超过40。
英文摘要：
Model-based reinforcement learning (RL) has shown great promise due to its sample efficiency, but still struggles with long-horizon sparse-reward tasks, especially in offline settings where the agent learns from a fixed dataset. We hypothesize that model-based RL agents struggle in these environments due to a lack of long-term planning capabilities, and that planning in a temporally abstract model of the environment can alleviate this issue. In this paper, we make two key contributions: 1) we introduce an offline model-based RL algorithm, IQL-TD-MPC, that extends the state-of-the-art Temporal Difference Learning for Model Predictive Control (TD-MPC) with Implicit Q-Learning (IQL); 2) we propose to use IQL-TD-MPC as a Manager in a hierarchical setting with any off-the-shelf offline RL algorithm as a Worker. More specifically, we pre-train a temporally abstract IQL-TD-MPC Manager to predict "intent embeddings", which roughly correspond to subgoals, via planning. We empirically show that augmenting state representations with intent embeddings generated by an IQL-TD-MPC manager significantly improves off-the-shelf offline RL agents' performance on some of the most challenging D4RL benchmark tasks. For instance, the offline RL algorithms AWAC, TD3-BC, DT, and CQL all get zero or near-zero normalized evaluation scores on the medium and large antmaze tasks, while our modification gives an average score over 40.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
