arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.03007cs.CLcs.AI
收藏

什么影响工具学习的稳定性?一项关于工具学习框架鲁棒性的实证研究

What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks

Shandong University(山东大学);University of Electronic Science and Technology of China(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Chengrui Huang, Zhengliang Shi, Yuntao Wen, Xiuying Chen, Peng Han, Shen Gao, Shuo Shang

AI总结:

本文通过实证研究系统分析了影响工具学习框架稳定性的内外部因素,并在两个基准数据集上验证了增加尝试与探索可显著提升大语言模型的工具使用效果。

摘要:

工具学习方法增强了大语言模型(LLMs)与现实世界应用交互的能力。许多现有工作通过微调LLMs或设计提示,使LLMs能够选择合适的工具并正确调用它们以满足用户需求。然而,先前工作观察到,工具学习的性能会因任务、数据集、训练设置和算法的不同而变化。如果不理解这些因素的影响,就可能导致结果不一致、模型部署效率低下以及工具利用不充分,最终阻碍LLMs在实际场景中的集成与可扩展性。因此,本文探讨了内部和外部因素对工具学习框架性能的影响。通过在两个基准数据集上的大量实验,我们发现了一些对未来工作具有启发意义的结论,包括观察到LLMs可以从增加的尝试和探索中显著受益。我们相信,我们的实证研究为未来的工具学习研究提供了新的视角。

英文摘要:

Tool learning methods have enhanced the ability of large language models (LLMs) to interact with real-world applications. Many existing works fine-tune LLMs or design prompts to enable LLMs to select appropriate tools and correctly invoke them to meet user requirements. However, it is observed in previous works that the performance of tool learning varies from tasks, datasets, training settings, and algorithms. Without understanding the impact of these factors, it can lead to inconsistent results, inefficient model deployment, and suboptimal tool utilization, ultimately hindering the practical integration and scalability of LLMs in real-world scenarios. Therefore, in this paper, we explore the impact of both internal and external factors on the performance of tool learning frameworks. Through extensive experiments on two benchmark datasets, we find several insightful conclusions for future work, including the observation that LLMs can benefit significantly from increased trial and exploration. We believe our empirical study provides a new perspective for future tool learning research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑