MERRY: 多模态情感与角色一致性评估的语义解耦框架
MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
AI总结：
MERRY提出一种语义解耦的评估框架，用于评估多模态角色扮演代理的情感与角色一致性，通过改进的指标和双向证据寻找任务提升评估效果。
摘要：
多模态角色扮演代理（MRPAs）因其能够提供更沉浸式的多模态情感交互能力而受到越来越多的关注。然而，现有研究仍然依赖纯文本基准来评估MRPAs的文本响应，而将多模态表达的评估 solely 委托给模态合成度量。这种评估范式一方面将语义评估与模态生成纠缠在一起，导致错误归因模糊；另一方面仍然受限于对人类判断的依赖。为此，我们提出MERRY，一种用于评估多模态情感与角色一致性的语义解耦评估框架。该框架引入了五种改进的EC指标和三种RC指标。值得注意的是，我们将传统的主观评分方法转变为一种新的双向证据寻找任务，显著提高了LLM-as-Judge评估的人类一致性。基于MERRY，我们进行了广泛的评估。我们的实证结果主要揭示了以下几点：（1）在合成数据集上训练倾向于降低情感一致性，而在真实世界数据集上训练则能提高它；（2）现有模型在情感模板化和简化方面存在问题，表现出对细粒度负面情绪的正偏和性能瓶颈；（3）简单的提示方法增强了弱模型但限制了强模型，而简单的微调方法则在角色泛化方面表现不佳。代码和数据集已提供。
英文摘要：
Multimodal Role-Playing Agents (MRPAs) are attracting increasing attention due to their ability to deliver more immersive multimodal emotional interactions. However, existing studies still rely on pure textual benchmarks to evaluate the text responses of MRPAs, while delegating the assessment of their multimodal expressions solely to modality-synthesis metrics. This evaluation paradigm, on the one hand, entangles semantic assessment with modality generation, leading to ambiguous error attribution, and on the other hand remains constrained by the heavy reliance on human judgment. To this end, we propose MERRY, a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents. This framework introduce five refined metrics for EC and three for RC. Notably, we transform the traditional subjective scoring approach into a novel bidirectional-evidence-finding task, significantly improving the human agreement of LLM-as-Judge evaluations. Based on MERRY, we conduct extensive evaluations. Our empirical results primarily reveal that: (1) Training on synthetic datasets tends to reduce emotional consistency, whereas training on real-world datasets improves it; (2) Existing models suffer from emotional templatization and simplification, exhibiting positive-bias and performance bottleneck in fine-grained negative emotions; (3) Simple prompting method strengthens the weak models but constrains the strong ones, while simple fine-tuning method suffers from poor role generalization. Codes and dataset are available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
