arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18277cs.LGcs.AIstat.ML
收藏

PRISM: 基于对称性的并行奖励整合用于多目标强化学习

PRISM: Parallel Reward Integration with Symmetry for MORL

University of Edinburgh(爱丁堡大学);Meta Superintelligence Labs(Meta超智能实验室)

Finn van der Knaap, Kejiang Qian, Zheng Xu, Fengxiang He

AI总结:

PRISM通过引入基于对称性的并行奖励整合算法,解决多目标强化学习中稀疏奖励与密集奖励之间的样本效率问题,提升Pareto覆盖和分布平衡。

摘要:

本文研究了异质多目标强化学习(MORL),其中目标在时间频率上可能差异显著。这种异质性允许密集目标主导学习,而稀疏长周期奖励则获得弱信用分配,导致样本效率低下。我们提出了一种基于对称性的并行奖励整合(PRISM)算法,通过在对齐奖励通道时施加反射对称性作为归纳偏置。PRISM引入了ReSymNet,一个理论驱动的模型,通过残差块学习一个缩放的机会价值,以加速探索同时保持最优策略。我们还提出了SymReg,一种反射等价性正则化器,强制智能体镜像并对政策搜索限制在反射等价子空间内。这种限制可以证明地减少假设复杂性并提高泛化能力。在MuJoCo基准测试中,PRISM在稀疏奖励基线和使用完整密集奖励训练的Oracle上均表现优异,提高了Pareto覆盖和分布平衡:它在基线上的超体积收益超过100%,在Oracle上的收益达到32%。代码见https://github.com/EVIEHub/PRISM。

英文摘要:

This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100\% over the baseline and up to 32\% over the oracle. The code is at \href{https://github.com/EVIEHub/PRISM}{https://github.com/EVIEHub/PRISM}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑