arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11359cs.AIcs.CL
收藏

以函数作为可学习权重的语言模型智能体离线训练

Offline Training of Language Model Agents with Functions as Learnable Weights

Pennsylvania State University(宾夕法尼亚州立大学);University of Washington(华盛顿大学);University of Southern California(南加州大学);Microsoft Research(微软研究院)

Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, Qingyun Wu

AI总结:

本文提出一种不修改LLM权重、而将函数作为可学习参数的智能体离线训练范式,并开发AgentOptimizer及回滚、早停策略,显著提升多种下游任务中LLM智能体的表现。

摘要:

研究人员和实践者最近将强大的大语言模型(LLMs)重新定义为智能体,使其能够主要通过使用专门函数来自动化复杂任务。为了促进LLM智能体的发展,我们提出了一种在不修改LLM权重的情况下训练LLM智能体的新范式,这在LLM难以修改或无法访问时尤其有用。受人类不断锻造工具以适应现实世界任务的启发,而不是改变我们的生物结构去适应一组静态工具,我们提出逐步锻造智能体的函数以更好地解决下游任务,而不是修改LLM权重。通过将函数视为可学习的“智能体参数”,并利用人工智能中模型训练的基本思想,我们开发了AgentOptimizer,它使用LLM来更新智能体的函数,并设计了一种具有两种策略(回滚和早停)的智能体训练算法,以简化训练过程。通过大量实验,我们展示了该智能体训练范式能够显著提高代表性LLM智能体在各种下游任务中的性能。我们还研究了智能体训练在学习曲线和领域可迁移性等方面的行为。

英文摘要:

Researchers and practitioners have recently reframed powerful Large Language Models (LLMs) as agents, enabling them to automate complex tasks largely via the use of specialized functions. To facilitate the development of LLM agents, we present a novel paradigm of training LLM agents without modifying the LLM weights, which is particularly useful when the LLMs are difficult or inaccessible for modifications. Inspired by how humans continuously forge tools to adapt to real-world tasks, rather than change our biological structure to fit a static set of tools, we propose to progressively forge agent's functions to better solve the downstream tasks instead of modifying the LLM weights. By treating the functions as learnable `agent parameters' and leveraging the fundamental idea of model training in artificial intelligence, we develop AgentOptimizer that employs the LLM to update agents' functions and devise an agent training algorithm with two strategies, roll-back, and early-stop, to streamline the training process. With extensive experiments, we showcase that the agent training paradigm could significantly improve the performance of representative LLM agents in various downstream tasks. We also study the behavior of the agent training regarding aspects like the learning curve and domain transferability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑