LLM与中文动漫爱好者：关于情感支持性角色扮演的对比研究
LLMs vs. Chinese Anime Enthusiasts: A Comparative Study on Emotionally Supportive Role-Playing
AI总结：
本文提出首个情感支持性角色扮演数据集ChatAnime，通过对比10个LLM与40名中文动漫爱好者的表现发现，顶级LLM在角色扮演和情感支持上超越人类，但人类在回复多样性上仍保持优势。
摘要：
大型语言模型（LLMs）在角色扮演对话和提供情感支持方面作为独立研究方向展现出了令人印象深刻的能力。然而，在结合这些能力以实现与虚拟角色的情感支持性互动方面，仍存在显著的研究空白。为了填补这一空白，我们以动漫角色为案例进行研究，因为它们具有鲜明的性格和庞大的粉丝群体。这一选择使我们能够有效评估LLMs在维持特定角色特征的同时提供情感支持的能力。我们引入了ChatAnime，这是首个情感支持性角色扮演（Emotionally Supportive Role-Playing, ESRP）数据集。我们首先从热门动漫社区精心挑选了20个顶级角色，并设计了60个以情感为中心的真实场景问题。接着，我们通过全国范围的选拔过程，筛选出40名对特定角色有深入了解且具有丰富角色扮演经验的中文动漫爱好者。随后，我们系统地收集了来自10个LLMs和这40名中文动漫爱好者的两轮对话数据。为了评估LLMs的ESRP表现，我们设计了一套以用户体验为导向的评估系统，包含跨越基础对话、角色扮演和情感支持三个维度的9项细粒度指标，以及一项用于衡量回复多样性的总体指标。该数据集总共包含2,400条人工撰写和24,000条LLM生成的回答，并有超过132,000条人工标注作为支撑。实验结果表明，表现最优的LLMs在角色扮演和情感支持方面超越了人类粉丝，而人类在回复多样性上依然领先。我们希望这项工作能为未来优化LLMs在ESRP方面的研究提供宝贵的资源和见解。我们的数据集可在 https://github.com/LanlanQiu/ChatAnime 获取。
英文摘要：
Large Language Models (LLMs) have demonstrated impressive capabilities in role-playing conversations and providing emotional support as separate research directions. However, there remains a significant research gap in combining these capabilities to enable emotionally supportive interactions with virtual characters. To address this research gap, we focus on anime characters as a case study because of their well-defined personalities and large fan bases. This choice enables us to effectively evaluate how well LLMs can provide emotional support while maintaining specific character traits. We introduce ChatAnime, the first Emotionally Supportive Role-Playing (ESRP) dataset. We first thoughtfully select 20 top-tier characters from popular anime communities and design 60 emotion-centric real-world scenario questions. Then, we execute a nationwide selection process to identify 40 Chinese anime enthusiasts with profound knowledge of specific characters and extensive experience in role-playing. Next, we systematically collect two rounds of dialogue data from 10 LLMs and these 40 Chinese anime enthusiasts. To evaluate the ESRP performance of LLMs, we design a user experience-oriented evaluation system featuring 9 fine-grained metrics across three dimensions: basic dialogue, role-playing and emotional support, along with an overall metric for response diversity. In total, the dataset comprises 2,400 human-written and 24,000 LLM-generated answers, supported by over 132,000 human annotations. Experimental results show that top-performing LLMs surpass human fans in role-playing and emotional support, while humans still lead in response diversity. We hope this work can provide valuable resources and insights for future research on optimizing LLMs in ESRP. Our datasets are available at https://github.com/LanlanQiu/ChatAnime.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
