CoSER：一个综合的文学数据集和框架，用于训练和评估LLM角色扮演和人设模拟
CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation
AI总结：
CoSER通过高质量文学数据集和开放模型，训练和评估LLM角色扮演能力，其70B模型在多个基准测试中超越GPT-4o。
摘要：
角色扮演语言代理（RPLAs）作为大语言模型（LLMs）的有前景应用已逐渐兴起。然而，模拟已建立的角色对RPLAs来说是一项具有挑战性的任务，这归因于缺乏真实角色数据集和使用此类数据的细致评估方法。在本文中，我们提出了CoSER，一个高质量数据集、开放模型和评估协议的集合，旨在有效训练和评估已建立角色的RPLAs。CoSER数据集涵盖771本著名书籍中的17,966个角色。它提供了具有真实世界复杂性的真实对话，以及各种数据类型，如对话设置、角色经历和内部思想。借鉴表演方法，我们引入了给定情境表演，用于训练和评估角色扮演LLMs，其中LLMs依次在书场景中扮演多个角色。使用我们的数据集，我们开发了CoSER 8B和CoSER 70B，即基于LLaMA-3.1模型的先进开放角色扮演LLMs。广泛的实验证明了CoSER数据集在RPLA训练、评估和检索中的价值。此外，CoSER 70B在我们的评估和三个现有基准测试中表现出最先进的性能，即在InCharacter和LifeChoice基准测试中分别达到75.80%和93.47%的准确率。
英文摘要：
Role-playing language agents (RPLAs) have emerged as promising applications of large language models (LLMs). However, simulating established characters presents a challenging task for RPLAs, due to the lack of authentic character datasets and nuanced evaluation methods using such data. In this paper, we present CoSER, a collection of a high-quality dataset, open models, and an evaluation protocol towards effective RPLAs of established characters. The CoSER dataset covers 17,966 characters from 771 renowned books. It provides authentic dialogues with real-world intricacies, as well as diverse data types such as conversation setups, character experiences and internal thoughts. Drawing from acting methodology, we introduce given-circumstance acting for training and evaluating role-playing LLMs, where LLMs sequentially portray multiple characters in book scenes. Using our dataset, we develop CoSER 8B and CoSER 70B, i.e., advanced open role-playing LLMs built on LLaMA-3.1 models. Extensive experiments demonstrate the value of the CoSER dataset for RPLA training, evaluation and retrieval. Moreover, CoSER 70B exhibits state-of-the-art performance surpassing or matching GPT-4o on our evaluation and three existing benchmarks, i.e., achieving 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
