从个性化和主动性视角评估个性化工具增强的大型语言模型
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
AI总结：
针对现有基准未同时关注文本生成个性化与直接工具使用的问题，提出ETAPP基准及沙箱环境、800个测试案例，采用基于关键点的LLM评估方法减轻偏差，评估主流LLM并探究工具调用策略与微调效果，为改进个性化LLM智能体提供见解。
摘要：
个性化工具利用对于大型语言模型（LLMs）在多工具交互场景中与用户偏好对齐至关重要。然而，当前大多数基准主要关注文本生成的个性化或直接工具使用，未同时考虑两者。本研究提出新基准ETAPP用于评估个性化工具调用，构建沙箱环境及包含800个测试案例的综合数据集，覆盖多样用户画像。为提升评估准确性，提出基于关键点的LLM评估方法：通过手动标注每个测试案例的关键点并提供给LLM作为参考，减轻LLM作为评判系统的偏差。此外，评估主流LLM并开展深入分析，还探究不同工具调用策略对LLM个性化性能的影响及微调在本任务中的效果。验证了偏好设置和基于关键点评估方法的有效性。研究发现为改进个性化LLM智能体提供见解，代码见https://github.com/hypasd-art/ETAPP。
英文摘要：
Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools. However, most of the current benchmarks primarily focus on either personalization of text generation or direct tool-utilizing, without considering both. In this work, we introduce a novel benchmark ETAPP for evaluating personalized tool invocation, establishing a sandbox environment, and a comprehensive dataset of 800 testing cases covering diverse user profiles. To improve the accuracy of our evaluation, we propose a key-point-based LLM evaluation method, mitigating biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference. Additionally, we evaluate the excellent LLMs and provide an in-depth analysis. Furthermore, we investigate the impact of different tool-invoking strategies on LLMs' personalization performance and the effects of fine-tuning in our task. The effectiveness of our preference-setting and key-point-based evaluation method is also validated. Our findings offer insights into improving personalized LLM agents. Our Code is available at https://github.com/hypasd-art/ETAPP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
