arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.05950cs.LGcs.AIcs.RO
收藏

通过任务特定动作校正实现高效多任务强化学习

Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correction

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)

Jinyuan Feng, Min Chen, Zhiqiang Pu, Tenghai Qiu, Jianqiang Yi

AI总结:

针对多任务强化学习中的任务冲突与负干扰问题,本文提出任务特定动作校正(TSAC)方法,通过共享策略与动作校正策略的分解、目标导向稀疏奖励及拉格朗日优化,在Meta-World基准上显著提升了样本效率和动作执行效果。

摘要:

多任务强化学习(MTRL)展现出增强机器人泛化能力的潜力,使其能够同时执行多个任务。然而,MTRL的性能仍可能受到任务间冲突和负干扰的影响。为促进高效的多任务强化学习,我们提出了任务特定动作校正(TSAC),这是一种通用且互补的方法,专为同时学习多个任务而设计。TSAC将策略学习分解为两个独立的策略:共享策略(SP)和动作校正策略(ACP)。为缓解SP中因过度关注特定任务细节而产生的冲突,ACP引入面向目标的稀疏奖励,使智能体能够采取长期视角并实现跨任务泛化。额外的奖励将原始问题转化为多目标MTRL问题。此外,为将多目标MTRL转化为单目标形式,TSAC为稀疏奖励分配一个虚拟期望预算,并采用拉格朗日方法将带约束的单目标优化转化为无约束优化。在Meta-World的MT10和MT50基准上进行的实验评估表明,TSAC优于现有的最先进方法,在样本效率和有效动作执行方面均取得了显著提升。

英文摘要:

Multi-task reinforcement learning (MTRL) demonstrate potential for enhancing the generalization of a robot, enabling it to perform multiple tasks concurrently. However, the performance of MTRL may still be susceptible to conflicts between tasks and negative interference. To facilitate efficient MTRL, we propose Task-Specific Action Correction (TSAC), a general and complementary approach designed for simultaneous learning of multiple tasks. TSAC decomposes policy learning into two separate policies: a shared policy (SP) and an action correction policy (ACP). To alleviate conflicts resulting from excessive focus on specific tasks' details in SP, ACP incorporates goal-oriented sparse rewards, enabling an agent to adopt a long-term perspective and achieve generalization across tasks. Additional rewards transform the original problem into a multi-objective MTRL problem. Furthermore, to convert the multi-objective MTRL into a single-objective formulation, TSAC assigns a virtual expected budget to the sparse rewards and employs Lagrangian method to transform a constrained single-objective optimization into an unconstrained one. Experimental evaluations conducted on Meta-World's MT10 and MT50 benchmarks demonstrate that TSAC outperforms existing state-of-the-art methods, achieving significant improvements in both sample efficiency and effective action execution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑