SocialBench:角色扮演对话智能体的社交性评估
SocialBench: Sociality Evaluation of Role-Playing Conversational Agents
AI总结:
提出SocialBench基准,从个体与群体层面评估角色扮演对话智能体的社交性,并通过主流开源和闭源LLM揭示群体互动中的能力差异与行为漂移。
摘要:
大型语言模型(LLM)推动了各类AI对话智能体的发展,其中包括模拟多样角色与人类行为的角色扮演对话智能体。尽管以往研究主要关注提升这些智能体的对话能力、角色特定知识和风格属性,但在评估其社交智能方面仍存在明显空白。本文提出SocialBench,这是首个旨在从个体和群体两个社会互动层面系统评估角色扮演对话智能体社交性的基准。该基准由多种来源构建,涵盖广泛的500个角色、超过6,000个问题提示以及30,800条多轮角色扮演话语。我们使用主流开源和闭源LLM在该基准上开展全面评估。我们发现,智能体在个体层面表现出色,并不意味着其在群体层面同样熟练。此外,由于群体中其他智能体施加的影响,个体行为可能发生漂移。SocialBench上的实验结果证实,其作为评估角色扮演对话智能体社会互动的测试平台具有重要意义。该基准可在https://github.com/X-PLUG/SocialBench公开获取。
英文摘要:
Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/SocialBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。