PersonaLens：对话式AI助手个性化评估基准
PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
AI总结：
针对现有对话AI个性化评估基准的不足，研究提出综合基准PersonaLens，含多样用户画像与专用LLM智能体，实验发现当前LLM助手个性化能力差异显著，为相关系统发展提供参考。
摘要：
大语言模型（LLMs）推动了对话式AI助手的发展。然而，系统性评估这些助手的个性化应用能力——即在完成任务时适配个体用户偏好的能力——仍具挑战。现有个性化基准聚焦闲聊、非对话任务或狭窄领域，无法捕捉个性化任务导向型助手的复杂性。为此，我们推出PersonaLens，这是一套用于评估任务导向型AI助手个性化能力的综合基准。该基准包含具备丰富偏好与交互历史的多样化用户画像，以及两个基于LLM的专用智能体：一个用户智能体（与AI助手开展真实的任务导向对话），一个评判智能体（采用LLM-as-a-Judge范式评估个性化程度、回复质量与任务完成度）。通过对当前主流LLM助手在多样化任务上的大量实验，我们发现其个性化能力存在显著差异，为对话式AI系统的发展提供了关键洞见。
英文摘要：
Large language models (LLMs) have advanced conversational AI assistants. However, systematically evaluating how well these assistants apply personalization--adapting to individual user preferences while completing tasks--remains challenging. Existing personalization benchmarks focus on chit-chat, non-conversational tasks, or narrow domains, failing to capture the complexities of personalized task-oriented assistance. To address this, we introduce PersonaLens, a comprehensive benchmark for evaluating personalization in task-oriented AI assistants. Our benchmark features diverse user profiles equipped with rich preferences and interaction histories, along with two specialized LLM-based agents: a user agent that engages in realistic task-oriented dialogues with AI assistants, and a judge agent that employs the LLM-as-a-Judge paradigm to assess personalization, response quality, and task success. Through extensive experiments with current LLM assistants across diverse tasks, we reveal significant variability in their personalization capabilities, providing crucial insights for advancing conversational AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
