arXiv 2410.031982026-02-04cs.CL

PersoBench：大型语言模型中个性化响应生成的基准测试

PersoBench: Benchmarking Personalized Response Generation in Large Language Models

School of Computing, Macquarie University（计算机学院，麦考瑞大学）；School of Electrical and Computer Engineering, Shiraz University（电气与计算机工程学院，谢里兹大学）

Saleh Afzoon, Zahra Jamali, Usman Naseem, Amin Beheshti

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

PersoBench通过结构化流程评估LLMs在零样本设置中生成个性化响应的能力，揭示其在流畅性、个性化和连贯性方面的不足。

摘要：

尽管大型语言模型（LLMs）在对话能力上表现出色，但其在生成个性化响应方面的能力仍不明确。尽管最近的基准测试利用LLM进行判断，自动评估角色扮演情境中的个性一致性，但响应生成中的个性化评估仍处于探索阶段。为填补这一空白，我们提出了一种自动化基准测试流程，PersoBench，用于评估LLMs在零样本设置中，在具有角色意识的对话生成中个性化能力。我们的框架采用了一个结构化的流程，包括说话者意识的注释、任务特定且上下文驱动的提示构建、响应后处理以及在生成质量的多个维度上进行自动化评估。特别是，该流程执行文本预处理和说话者标注，构建包含任务指示和LLM角色的结构化提示，验证响应格式，并在流畅性、个性化、多样性和连贯性等方面评估有效的输出。我们使用知名数据集和一系列明确的指标，评估了四个开源和四个闭源LLMs的性能。我们的发现表明，尽管LLMs在生成流畅和多样化的响应方面表现出色，但他们在考虑对话上下文和提供的角色时，生成个性化和连贯的响应方面还远远不够满意。

英文摘要：

While large language models (LLMs) have exhibited impressive conversational capabilities, their proficiency in delivering personalized responses remains unclear. Although recent benchmarks automatically evaluate persona consistency in role-playing contexts using LLM-based judgment, the evaluation of personalization in response generation remains underexplored. To address this gap, we present an automated benchmarking pipeline, PersoBench, to evaluate the personalization ability of LLMs in persona-aware dialogue generation within a zero-shot setting. Our framework employs a structured pipeline comprising speaker-aware annotation, task-specific and context-driven prompt construction, response post-processing, and automated evaluation across multiple dimensions of generation quality. In particular, the pipeline performs text preprocessing and speaker labeling, constructs structured prompts with task instructions and LLM roles, validates response format, and evaluates valid outputs across fluency, personalization, diversity, and coherence. We assess the performance of four open-source and four closed-source LLMs using well-known datasets and a range of explicit metrics. Our findings reveal that while LLMs excel at generating fluent and diverse responses, they are far from satisfactory in delivering personalized and coherent responses, considering both the conversation context and the provided personas.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。