高效利用当代大语言模型(LLMs)的能力正变得越来越具有挑战性,尤其是在直接微调成本高昂且往往不切实际的情况下。现有的免训练方法,包括人工设计或自动化设计的工作流,通常需要大量的人力投入,或者只能产生次优结果。本文提出了以弱驭强框架(Weak-for-Strong Harnessing,W4S),这是一个新颖的框架,通过定制规模更小、成本效益更高的语言模型来设计和优化用于驾驭更强模型的工作流。W4S将工作流设计形式化为一个多轮马尔可夫决策过程,并引入面向智能体工作流优化的强化学习(reinforcement learning for agentic workflow optimization,RLAO)来训练一个弱元智能体。通过与环境进行迭代交互,该元智能体学会在无需人工干预的情况下设计出越来越有效的工作流。实验结果表明了W4S的优越性:我们的7B元智能体仅用一小时的GPU训练时间,就在十一个基准测试上以2.9%至24.6%的幅度超越了最强基线,成功提升了GPT-3.5-Turbo和GPT-4o等最先进模型的性能。值得注意的是,W4S在已见和未见任务上均展现出强大的泛化能力,为直接微调强模型提供了一种高效且高性能的替代方案。
英文摘要:
Efficiently leveraging of the capabilities of contemporary large language models (LLMs) is increasingly challenging, particularly when direct fine-tuning is expensive and often impractical. Existing training-free methods, including manually or automated designed workflows, typically demand substantial human effort or yield suboptimal results. This paper proposes Weak-for-Strong Harnessing (W4S), a novel framework that customizes smaller, cost-efficient language models to design and optimize workflows for harnessing stronger models. W4S formulates workflow design as a multi-turn markov decision process and introduces reinforcement learning for agentic workflow optimization (RLAO) to train a weak meta-agent. Through iterative interaction with the environment, the meta-agent learns to design increasingly effective workflows without manual intervention. Empirical results demonstrate the superiority of W4S that our 7B meta-agent, trained with just one GPU hour, outperforms the strongest baseline by 2.9% ~ 24.6% across eleven benchmarks, successfully elevating the performance of state-of-the-art models such as GPT-3.5-Turbo and GPT-4o. Notably, W4S exhibits strong generalization capabilities across both seen and unseen tasks, offering an efficient, high-performing alternative to directly fine-tuning strong models.
AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design
Meituan(美团);MBZUAI(Mohamed bin Zayed University of Artificial Intelligence);Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)
A Blind Trust, the Bloody Thrust: When Attacker-Controlled Hook Updates Steer AI Agent Harnesses towards Malicious Behaviors
Beijing University of Posts and Telecommunications(北京邮电大学);Data Technology Support Center of the Cyberspace Administration of China(中央网络安全和信息化委员会办公室数据技术支持中心);Beihang University(北京航空航天大学);Zhejiang University(浙江大学)
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DeepWisdom;Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Université de Montréal & Mila(蒙特利尔大学及Mila)
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Chinese University of Hong Kong(香港中文大学);Nanyang Technological University(南洋理工大学);Fudan University(复旦大学);National University of Singapore(新加坡国立大学);DeepWisdom(深度智慧)
MetaCaster: Meta-Harness-Optimized Agent for End-to-End Few-Shot Learning of Lightweight Time Series Forecasters
University of Houston(休斯顿大学);NEC Labs(NEC实验室);University of Waterloo(滑铁卢大学);University of Connecticut(康涅狄格大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Singapore Management University(新加坡管理大学)
RSIFlow: Adaptive Self-Improvement Workflows with an Evolving Meta-Agent
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(未来区块链与隐私计算北京高精尖创新中心);School of Artificial Intelligence(人工智能学院);Beihang University(北京航空航天大学);Tsinghua University(清华大学);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院)
MetaEvo: A Meta-Optimization Framework for Experience-Driven Agent Evolution
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Beijing Institute of Technology Southeast Academy of Information Technology(北京理工大学东南信息技术研究院)
GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tianjin University(天津大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));East China Normal University(华东师范大学);Shenzhen Research Institute of Big Data(深圳市大数据研究院)