arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.06101cs.AIcs.MA
收藏

与另一个你共同进化:使用序列协作多智能体强化学习微调大语言模型

Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning

School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Alibaba (China) Co., Ltd.(阿里巴巴(中国)有限公司);Macau University of Science and Technology(澳门科技大学)

Hao Ma, Tianyi Hu, Zhiqiang Pu, Boyin Liu, Xiaolin Ai, Yanyan Liang, Min Chen

AI总结:

针对现有RL微调LLM依赖PPO易出现次优表现和分布崩溃的问题,提出序列协作多智能体框架CORY,通过双智能体角色互换协同进化,实验验证其在策略最优性等方面优于PPO。

摘要:

强化学习(RL)已成为针对特定任务微调大语言模型(LLM)的关键技术。然而,主流的RL微调方法主要依赖PPO及其变体。尽管这些算法在通用RL场景中有效,但应用于LLM微调时,往往表现出次优性能,且容易出现分布崩溃问题。本文提出CORY(协同进化式LLM微调框架),将LLM的RL微调拓展为序列协作多智能体强化学习框架,以利用多智能体系统固有的协同进化与涌现能力。在CORY中,待微调的LLM首先被复制为两个自主智能体:先行者与观察者。先行者基于查询生成回复,观察者则结合查询与先行者的回复生成回复,二者共同训练。训练过程中,智能体定期交换角色,促进彼此间的协作与共同进化。实验通过在IMDB Review和GSM8K数据集上分别采用主观与客观奖励函数微调GPT-2和Llama-2,评估CORY的性能。结果表明,CORY在策略最优性、抗分布崩溃能力和训练鲁棒性方面均优于PPO,凸显了其作为现实场景中LLM优化方法的潜力。

英文摘要:

Reinforcement learning (RL) has emerged as a pivotal technique for fine-tuning large language models (LLMs) on specific tasks. However, prevailing RL fine-tuning methods predominantly rely on PPO and its variants. Though these algorithms are effective in general RL settings, they often exhibit suboptimal performance and vulnerability to distribution collapse when applied to the fine-tuning of LLMs. In this paper, we propose CORY, extending the RL fine-tuning of LLMs to a sequential cooperative multi-agent reinforcement learning framework, to leverage the inherent coevolution and emergent capabilities of multi-agent systems. In CORY, the LLM to be fine-tuned is initially duplicated into two autonomous agents: a pioneer and an observer. The pioneer generates responses based on queries, while the observer generates responses using both the queries and the pioneer's responses. The two agents are trained together. During training, the agents exchange roles periodically, fostering cooperation and coevolution between them. Experiments evaluate CORY's performance by fine-tuning GPT-2 and Llama-2 under subjective and objective reward functions on the IMDB Review and GSM8K datasets, respectively. Results show that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness, thereby underscoring its potential as a superior methodology for refining LLMs in real-world applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑