arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.04969cs.CLcs.AI
收藏

MPCEval: 多方对话生成的评估基准

MPCEval: A Benchmark for Multi-Party Conversation Generation

Duke University(杜克大学);Tanka AI(塔纳人工智能)

Minxing Zhang, Yi Yang, Zhuofan Jia, Xuan Yang, Jian Pei, Yuchen Zang, Xingwang Deng, Xianglong Chen

AI总结:

MPCEval提出一个多任务意识的评估和基准测试套件,用于评估多方对话生成的质量,揭示模型在参与平衡、内容进展和一致性方面的系统性特征。

摘要:

多方对话生成,如智能回复和协作助手,是生成式AI日益重要的能力,但其评估仍是一个关键瓶颈。与双边对话相比,多方设置引入了不同的挑战,包括复杂的轮流发言、角色依赖的说话者行为、长距离对话结构以及多个同样有效的延续。因此,我们引入MPCEval,一个多任务意识的评估和基准测试套件,用于多方对话生成。MPCEval将生成质量分解为说话者建模、内容质量以及说话者-内容一致性,并明确区分局部下一轮预测与全局完整对话生成。它提供了新颖的、定量的、参考自由且可重复的度量标准,这些度量标准可扩展到数据集和模型。我们应用MPCEval到多样化的公共和现实世界数据集,并评估现代生成方法与人类撰写的对话。结果揭示了参与平衡、内容进展和新颖性以及说话者-内容一致性方面的系统性、维度特定的模型特征,证明了评估目标对模型评估至关重要,且单一评分评估掩盖了多方对话行为的基本差异。MPCEval的实现及相关的评估代码已在https://github.com/Owen-Yang-18/MPCEval公开。

英文摘要:

Multi-party conversation generation, such as smart reply and collaborative assistants, is an increasingly important capability of generative AI, yet its evaluation remains a critical bottleneck. Compared to two-party dialogue, multi-party settings introduce distinct challenges, including complex turn-taking, role-dependent speaker behavior, long-range conversational structure, and multiple equally valid continuations. Accordingly, we introduce MPCEval, a task-aware evaluation and benchmarking suite for multi-party conversation generation. MPCEval decomposes generation quality into speaker modeling, content quality, and speaker--content consistency, and explicitly distinguishes local next-turn prediction from global full-conversation generation. It provides novel, quantitative, reference-free, and reproducible metrics that scale across datasets and models. We apply MPCEval to diverse public and real-world datasets and evaluate modern generation methods alongside human-authored conversations. The results reveal systematic, dimension-specific model characteristics in participation balance, content progression and novelty, and speaker--content consistency, demonstrating that evaluation objectives critically shape model assessment and that single-score evaluation obscures fundamental differences in multi-party conversational behavior. The implementation of MPCEval and the associated evaluation code are publicly available at https://github.com/Owen-Yang-18/MPCEval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑