arXiv 2505.131762025-05-22cs.CLcs.AI

ToolSpectrum：面向大型语言模型的个性化工具利用

ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models

School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；University of Science and Technology Beijing（北京科技大学）；The Chinese University of Hong Kong（香港中文大学）；Baidu Inc.（百度公司）

Zihao Cheng, Hongru Wang, Zeming Liu, Yuhang Guo, Yuanfang Guo, Yunhong Wang, Haifeng Wang

备注：: Accepted by ACL 2025 Findings

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有工具选择忽视上下文感知个性化的问题，提出ToolSpectrum基准，形式化用户画像与环境因素两个维度，实验表明个性化利用可提升体验，但现有LLM联合推理能力有限。

摘要：

将外部工具集成到大型语言模型（LLMs）中，虽然增强了其获取实时信息和特定领域服务的能力，但现有方法仅狭隘地关注根据用户指令进行功能性工具选择，忽视了工具选择中的上下文感知个性化。这种忽视导致用户满意度欠佳和工具利用效率低下，尤其是在工具集重叠且需要基于上下文因素进行细致选择的情况下。为弥补这一差距，我们引入了ToolSpectrum，一个旨在评估LLMs在个性化工具利用方面能力的基准。具体而言，我们形式化了两个关键个性化维度：用户画像和环境因素，并分析了它们对工具利用的单独及协同影响。通过在ToolSpectrum上的广泛实验，我们证明个性化工具利用在多种场景下显著提升了用户体验。然而，即使是最先进的LLMs也表现出在联合推理用户画像和环境因素方面的能力有限，常常偏向一个维度而牺牲另一个维度。我们的发现强调了在工具增强型LLMs中上下文感知个性化的必要性，并揭示了当前模型的关键局限性。我们的数据和代码可在https://github.com/Chengziha0/ToolSpectrum获取。

英文摘要：

While integrating external tools into large language models (LLMs) enhances their ability to access real-time information and domain-specific services, existing approaches focus narrowly on functional tool selection following user instructions, overlooking the context-aware personalization in tool selection. This oversight leads to suboptimal user satisfaction and inefficient tool utilization, particularly when overlapping toolsets require nuanced selection based on contextual factors. To bridge this gap, we introduce ToolSpectrum, a benchmark designed to evaluate LLMs' capabilities in personalized tool utilization. Specifically, we formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool utilization. Through extensive experiments on ToolSpectrum, we demonstrate that personalized tool utilization significantly improves user experience across diverse scenarios. However, even state-of-the-art LLMs exhibit the limited ability to reason jointly about user profiles and environmental factors, often prioritizing one dimension at the expense of the other. Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models. Our data and code are available at https://github.com/Chengziha0/ToolSpectrum.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。