回顾偏好回放改进偏好条件多目标强化学习
Hindsight Preference Replay Improves Preference-Conditioned Multi-Objective Reinforcement Learning
AI总结：
HPR通过回放增强策略改进CAPQL，提升多目标强化学习中偏好条件下的超体积和预期效用性能。
摘要：
多目标强化学习（MORL）使智能体能够在尊重用户偏好的情况下优化向量奖励。CAPQL，一种偏好条件的Actor-Critic方法，通过条件于权重向量w并限制数据使用到特定偏好下收集的数据，从而实现这一目标，不使用其他偏好下的离策略数据。我们引入了回顾偏好回放（HPR），一种简单且通用的回放增强策略，通过回溯性地将存储的转移重新标记为替代偏好。这在不改变CAPQL架构或损失函数的情况下，增强了偏好单纯形上的监督密度。在六个MO-Gymnasium运动任务上，使用预期效用（EUM）、超体积（HV）和稀疏性，在固定300000步预算下评估HPR-CAPQL，在六个环境中中的五个提升了HV，在四个环境中提升了EUM。例如，在mo-humanoid-v5中，EUM从323±125增加到1613±464，HV从0.52M增加到9.63M，具有强统计支持。mo-halfcheetah-v5仍然是一个挑战例外，其中CAPQL在可比的EUM下达到了更高的HV。我们报告了所有任务的最终总结和帕累托前沿可视化。
英文摘要：
Multi-objective reinforcement learning (MORL) enables agents to optimize vector-valued rewards while respecting user preferences. CAPQL, a preference-conditioned actor-critic method, achieves this by conditioning on weight vectors w and restricts data usage to the specific preferences under which it was collected, leaving off-policy data from other preferences unused. We introduce Hindsight Preference Replay (HPR), a simple and general replay augmentation strategy that retroactively relabels stored transitions with alternative preferences. This densifies supervision across the preference simplex without altering the CAPQL architecture or loss functions. Evaluated on six MO-Gymnasium locomotion tasks at a fixed 300000-step budget using expected utility (EUM), hypervolume (HV), and sparsity, HPR-CAPQL improves HV in five of six environments and EUM in four of six. On mo-humanoid-v5, for instance, EUM rises from $323\!\pm\!125$ to $1613\!\pm\!464$ and HV from 0.52M to 9.63M, with strong statistical support. mo-halfcheetah-v5 remains a challenging exception where CAPQL attains higher HV at comparable EUM. We report final summaries and Pareto-front visualizations across all tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
