arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.18752cs.CVcs.CL
收藏

GPT4Tools:通过自指令教会大型语言模型使用工具

GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Tencent AI Lab(腾讯人工智能实验室);Chinese University of Hong Kong(香港中文大学)

Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, Ying Shan

AI总结:

本文提出GPT4Tools,通过自指令生成数据集并结合LoRA优化,使开源LLM具备多模态工具使用能力,显著提升已见工具调用准确性并支持未见工具的零样本调用。

摘要:

本文旨在高效地使大型语言模型(LLMs)能够使用多模态工具。先进的自有LLMs,如ChatGPT和GPT-4,通过复杂的提示工程已展现出使用工具的巨大潜力。然而,这些模型通常依赖高昂的计算成本,且数据无法公开获取。为应对这些挑战,我们提出了基于自指令的GPT4Tools,使开源LLMs(如LLaMA和OPT)能够使用工具。该方法通过向先进的教师模型提示各种多模态上下文来生成指令遵循数据集。通过使用低秩适应(LoRA)优化,我们的方法帮助开源LLMs解决一系列视觉问题,包括视觉理解和图像生成。此外,我们提供了一个基准来评估LLMs使用工具的能力,该基准在零样本和微调两种方式下进行。大量实验证明了我们的方法在各种语言模型上的有效性,不仅显著提高了调用已见工具的准确性,还使模型具备了对未见工具的零样本能力。代码和演示可在https://github.com/StevenGrove/GPT4Tools获取。

英文摘要:

This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools. It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools. The code and demo are available at https://github.com/StevenGrove/GPT4Tools.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑