Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Alibaba (China) Co., Ltd.(阿里巴巴(中国)有限公司);Macau University of Science and Technology(澳门科技大学)
Reinforcement learning (RL) has emerged as a pivotal technique for fine-tuning large language models (LLMs) on specific tasks. However, prevailing RL fine-tuning methods predominantly rely on PPO and its variants. Though these algorithms are effective in general RL settings, they often exhibit suboptimal performance and vulnerability to distribution collapse when applied to the fine-tuning of LLMs. In this paper, we propose CORY, extending the RL fine-tuning of LLMs to a sequential cooperative multi-agent reinforcement learning framework, to leverage the inherent coevolution and emergent capabilities of multi-agent systems. In CORY, the LLM to be fine-tuned is initially duplicated into two autonomous agents: a pioneer and an observer. The pioneer generates responses based on queries, while the observer generates responses using both the queries and the pioneer's responses. The two agents are trained together. During training, the agents exchange roles periodically, fostering cooperation and coevolution between them. Experiments evaluate CORY's performance by fine-tuning GPT-2 and Llama-2 under subjective and objective reward functions on the IMDB Review and GSM8K datasets, respectively. Results show that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness, thereby underscoring its potential as a superior methodology for refining LLMs in real-world applications.
Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
针对多智能体同时行动难以形成均衡的问题,本文构建时空序贯决策结构,并用共享条件超网络实现 N 级 Stackelberg 策略学习,在矩阵博弈、合作与混合任务中验证了有效性。
A Multi-Agent Reinforcement Learning Approach for Cooperative Air-Ground-Human Crowdsensing in Emergency Rescue
College of Systems Engineering, National University of Defense Technology(系统工程学院,国防科技大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所)
V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));The University of Hong Kong(香港大学);University of Chinese Academy of Science(中国科学院大学);University of Science and Technology Beijing(北京科技大学);Beijing Institute of Technology(北京理工大学);Baidu Inc.(百度公司);Beijing Connected and Autonomous Vehicles Technology Co., Ltd(北京车网互联科技有限公司)
Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework
Washington University in St. Louis(华盛顿大学圣路易斯分校);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Analysing the behaviour of robot teams through relational sequential pattern mining
University of Bari “Aldo Moro”(巴里大学“阿尔多·莫罗”);Imperial College(帝国理工学院);IIIA - Artificial Intelligence Research Institute, CSIC - Spanish Council for Scientific Research(西班牙国家研究委员会人工智能研究所(IIIA))