arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.04785cs.AI
收藏

以弱驭强:训练弱元智能体以驾驭强执行器

Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors

Stanford University(斯坦福大学);EPFL(洛桑联邦理工学院);UNC-Chapel Hill(北卡罗来纳大学教堂山分校)

Fan Nie, Lan Feng, Haotian Ye, Weixin Liang, Pan Lu, Huaxiu Yao, Alexandre Alahi, James Zou

AI总结:

本文提出以弱驭强框架W4S,将工作流设计建模为多轮马尔可夫决策过程,并用强化学习训练7B弱元智能体,使其以极低成本自动设计工作流来驾驭GPT-4o等强模型,在多个基准上显著超越基线并具备良好泛化能力。

摘要:

高效利用当代大语言模型(LLMs)的能力正变得越来越具有挑战性,尤其是在直接微调成本高昂且往往不切实际的情况下。现有的免训练方法,包括人工设计或自动化设计的工作流,通常需要大量的人力投入,或者只能产生次优结果。本文提出了以弱驭强框架(Weak-for-Strong Harnessing,W4S),这是一个新颖的框架,通过定制规模更小、成本效益更高的语言模型来设计和优化用于驾驭更强模型的工作流。W4S将工作流设计形式化为一个多轮马尔可夫决策过程,并引入面向智能体工作流优化的强化学习(reinforcement learning for agentic workflow optimization,RLAO)来训练一个弱元智能体。通过与环境进行迭代交互,该元智能体学会在无需人工干预的情况下设计出越来越有效的工作流。实验结果表明了W4S的优越性:我们的7B元智能体仅用一小时的GPU训练时间,就在十一个基准测试上以2.9%至24.6%的幅度超越了最强基线,成功提升了GPT-3.5-Turbo和GPT-4o等最先进模型的性能。值得注意的是,W4S在已见和未见任务上均展现出强大的泛化能力,为直接微调强模型提供了一种高效且高性能的替代方案。

英文摘要:

Efficiently leveraging of the capabilities of contemporary large language models (LLMs) is increasingly challenging, particularly when direct fine-tuning is expensive and often impractical. Existing training-free methods, including manually or automated designed workflows, typically demand substantial human effort or yield suboptimal results. This paper proposes Weak-for-Strong Harnessing (W4S), a novel framework that customizes smaller, cost-efficient language models to design and optimize workflows for harnessing stronger models. W4S formulates workflow design as a multi-turn markov decision process and introduces reinforcement learning for agentic workflow optimization (RLAO) to train a weak meta-agent. Through iterative interaction with the environment, the meta-agent learns to design increasingly effective workflows without manual intervention. Empirical results demonstrate the superiority of W4S that our 7B meta-agent, trained with just one GPU hour, outperforms the strongest baseline by 2.9% ~ 24.6% across eleven benchmarks, successfully elevating the performance of state-of-the-art models such as GPT-3.5-Turbo and GPT-4o. Notably, W4S exhibits strong generalization capabilities across both seen and unseen tasks, offering an efficient, high-performing alternative to directly fine-tuning strong models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑