arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.09706cs.LG
收藏

通过强化学习训练一个模型以掌握跨层级的代理行为

Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning

Peking University(北京大学);National University of Singapore(新加坡国立大学)

Kaichen He, Zihao Wang, Muyao Li, Anji Liu, Yitao Liang

AI总结:

本文提出CrossHA,一种统一的代理模型,能够掌握异构的动作空间并自主选择每一步轨迹中最有效的接口,通过结合冷启动监督微调和多轮组相对策略优化(GRPO)算法,实现适应性动作切换,在Minecraft开放世界中超过800个任务上展示了最先进的性能。

摘要:

代理AI的范式正从工程复杂的流程转向训练后的原生模型。然而,现有代理通常局限于静态的预定义动作空间,如仅使用API、GUI事件或机器人命令。这种刚性限制了它们在动态环境中适应性,其中最佳交互粒度会根据情境变化而变化。为弥合这一差距,我们提出了CrossHA,一种统一的代理模型,能够掌握异构的动作空间并自主选择每一步轨迹中最有效的接口。我们引入了一个全面的训练管道,整合了冷启动监督微调与多轮组相对策略优化(GRPO)算法。该方法使代理能够学习适应性动作切换,平衡高层效率与低层精度,而无需人为指定规则。在Minecraft开放世界中超过800个任务的广泛实验表明,CrossHA实现了最先进的性能。通过动态利用多样化的动作空间,我们的模型显著优于固定动作基线,展现出在长时间推理中的优越泛化能力和效率。所有代码和模型均在https://github.com/CraftJarvis/OpenHA上提供。

英文摘要:

The paradigm of agentic AI is shifting from engineered complex workflows to post-training native models. However, existing agents are typically confined to static, predefined action spaces-such as exclusively using APIs, GUI events, or robotic commands. This rigidity limits their adaptability in dynamic environments where the optimal granularity of interaction varies contextually. To bridge this gap, we propose CrossHA, a unified agentic model that masters heterogeneous action spaces and autonomously selects the most effective interface for each step of a trajectory. We introduce a comprehensive training pipeline that integrates cold-start supervised fine-tuning with a Multi-Turn Group Relative Policy Optimization (GRPO) algorithm. This approach enables the agent to learn adaptive action switching-balancing high-level efficiency with low-level precision-without human-specified rules. Extensive experiments on over 800 tasks in the open-world Minecraft environment demonstrate that CrossHA achieves state-of-the-art performance. By dynamically leveraging the strengths of diverse action spaces, our model significantly outperforms fixed-action baselines, exhibiting superior generalization and efficiency in long-horizon reasoning. All code and models are available at https://github.com/CraftJarvis/OpenHA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑