arXiv 2507.203522025-10-23cs.CL

RMTBench：通过多轮用户中心角色扮演对大型语言模型进行基准测试

RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；Alibaba Group（阿里巴巴集团）；University of Chinese Academy of Sciences（中国科学院大学）

Hao Xiang, Tianyi Tang, Yang Su, Bowen Yu, An Yang, Fei Huang, Yichang Zhang, Yaojie Lu, Hongyu Lin, Xianpei Han, Jingren Zhou, Junyang Lin, Le Sun

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有角色扮演基准忽视用户意图的问题，提出用户中心双语基准RMTBench，含80个角色和8000余轮对话，基于用户动机构建对话并采用多轮模拟与LLM评分，弥合学术评估与实际部署差距。

摘要：

大型语言模型（LLMs）的最新进展显示出其在角色扮演应用中的巨大潜力。评估这些能力变得至关重要，但仍具挑战性。现有基准大多采用“角色中心”方法，将用户与角色的交互简化为孤立的问答任务，未能反映现实应用。为解决这一局限，我们引入了RMTBench，一个全面的“用户中心”双语角色扮演基准，包含80个多样化的角色和超过8000轮对话。RMTBench包括具有详细背景的自定义角色和由简单特征定义的抽象角色，能够在各种用户场景下进行评估。我们的基准基于明确的用户动机而非角色描述构建对话，确保与实际用户应用的一致性。此外，我们构建了一个真实的多轮对话模拟机制。通过精心选择的评估维度和基于LLM的评分，该机制捕捉用户与角色之间对话的复杂意图。通过将焦点从角色背景转移到用户意图的实现，RMTBench弥合了学术评估与实际部署需求之间的差距，为评估LLMs中的角色扮演能力提供了一个更有效的框架。所有代码和数据集将很快发布。我们在https://huggingface.co/datasets/xiangh/RMTBENCH发布数据集。

英文摘要：

Recent advancements in Large Language Models (LLMs) have shown outstanding potential for role-playing applications. Evaluating these capabilities is becoming crucial yet remains challenging. Existing benchmarks mostly adopt a \textbf{character-centric} approach, simplify user-character interactions to isolated Q&A tasks, and fail to reflect real-world applications. To address this limitation, we introduce RMTBench, a comprehensive \textbf{user-centric} bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. RMTBench includes custom characters with detailed backgrounds and abstract characters defined by simple traits, enabling evaluation across various user scenarios. Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications. Furthermore, we construct an authentic multi-turn dialogue simulation mechanism. With carefully selected evaluation dimensions and LLM-based scoring, this mechanism captures the complex intention of conversations between the user and the character. By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements, offering a more effective framework for assessing role-playing capabilities in LLMs. All code and datasets will be released soon. We release the datasets at https://huggingface.co/datasets/xiangh/RMTBENCH.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。