评估LLM生成与人类撰写的角色扮演对话回应
Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialogues
AI总结：
本研究通过人类评估和LLM-as-a-judge方法，发现LLM在多轮角色扮演对话中回应质量随时间下降，而人类回应逐步提升，并提出了验证的混合评估框架。
摘要：
在长篇、基于知识的角色扮演对话中评估大型语言模型（LLMs）仍然具有挑战性。本研究通过人类评估（N=38）和自动化LLM-as-a-judge评估，比较了多轮专业培训模拟中LLM生成与人类撰写的回应。人类评估显示，LLM生成的回应质量在轮次间显著下降，尤其在自然性、上下文维持和整体质量方面，而人类撰写的回应则逐步改善。与此发现一致，参与者还表示对人类撰写的对话有一致的偏好。这些人类判断得到了我们自动化LLM-as-a-judge评估的验证，其中Gemini 2.0 Flash在零样本成对偏好和随机6样本构念评分上与人类评估者实现了强一致性，确认了LLM与人类回应之间质量差距随时间扩大。我们的工作贡献了一个多轮基准，揭示了LLM在基于知识的角色扮演对话中的性能退化，并提供了一个经过验证的混合评估框架，以指导LLM在培训模拟中的可靠集成。
英文摘要：
Evaluating large language models (LLMs) in long-form, knowledge-grounded role-play dialogues remains challenging. This study compares LLM-generated and human-authored responses in multi-turn professional training simulations through human evaluation ($N=38$) and automated LLM-as-a-judge assessment. Human evaluation revealed significant degradation in LLM-generated response quality across turns, particularly in naturalness, context maintenance and overall quality, while human-authored responses progressively improved. In line with this finding, participants also indicated a consistent preference for human-authored dialogue. These human judgements were validated by our automated LLM-as-a-judge evaluation, where Gemini 2.0 Flash achieved strong alignment with human evaluators on both zero-shot pairwise preference and stochastic 6-shot construct ratings, confirming the widening quality gap between LLM and human responses over time. Our work contributes a multi-turn benchmark exposing LLM degradation in knowledge-grounded role-play dialogues and provides a validated hybrid evaluation framework to guide the reliable integration of LLMs in training simulations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
