HALO：通过异质智能体李雅普诺夫策略优化学习人机协作
HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization
AI总结：
针对人机协作中人类行为多样性和环境变化导致的泛化与鲁棒性问题，提出异质智能体李雅普诺夫策略优化（HALO）框架，通过李雅普诺夫收缩稳定去中心化多智能体强化学习，并利用最优二次投影修正梯度，实现理性差距的单调收缩，提升协作性能。
摘要：
为了提高人机协作（HRC）的泛化性和韧性，机器人必须应对人类行为和情境的多种组合，这推动了多智能体强化学习（MARL）的应用。然而，机器人与人类之间的固有异质性造成了理性差距（RG），使得去中心化的策略更新偏离了合作联合优化。由此产生的学习问题是一个一般和可微博弈，因此独立的策略梯度更新在没有额外结构的情况下可能会振荡或发散。我们提出了异质智能体李雅普诺夫策略优化（HALO），这是一个通过强制策略参数空间中的李雅普诺夫收缩来稳定去中心化MARL的框架。与针对约束马尔可夫决策过程中状态/轨迹约束的基于李雅普诺夫的安全RL不同，HALO使用李雅普诺夫认证来稳定去中心化策略学习。HALO通过最优二次投影修正去中心化梯度，确保RG的单调收缩，并实现对开放式交互空间的有效探索。大量的仿真和真实人形机器人实验表明，这种认证的稳定性提高了协作边缘情况下的泛化性和鲁棒性。我们的项目网站位于https://HaoZhang-THU.github.io/HALO/。
英文摘要：
To improve generalization and resilience in human-robot collaboration (HRC), robots must contend with diverse combinations of human behaviors and contexts, motivating multi-agent reinforcement learning (MARL). However, inherent heterogeneity between robots and humans creates a rationality gap (RG), where decentralized policy updates deviate from cooperative joint optimization. The resulting learning problem is a general-sum differentiable game, so independent policy-gradient updates can oscillate or diverge without added structure. We propose heterogeneous-agent Lyapunov policy optimization (HALO), a framework that stabilizes decentralized MARL by enforcing Lyapunov-based contraction in policy-parameter space. Unlike Lyapunov-based safe RL, which targets state/trajectory constraints in constrained Markov decision processes, HALO uses Lyapunov certification to stabilize decentralized policy learning. HALO rectifies decentralized gradients via optimal quadratic projections, ensuring monotonic contraction of RG and enabling effective exploration of open-ended interaction spaces. Extensive simulations and real-world humanoid-robot experiments show that this certified stability improves generalization and robustness in collaborative corner cases. Our project website is available at https://HaoZhang-THU.github.io/HALO/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
