MEDAL：一个用于评估LLM作为多语言开放领域对话评估器的框架
MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
AI总结：
MEDAL框架通过多语言多代理方法，评估LLM作为开放领域对话评估者的性能，揭示了现有评判者在检测细微问题上的不足。
摘要：
评估开放领域聊天机器人的质量越来越依赖于LLM作为自动评判者的作用。然而，现有的元评估基准是静态的、过时的，并且缺乏多语言覆盖，限制了它们全面捕捉评估中细微弱点的能力。我们介绍了MEDAL，一个自动化多代理框架，用于整理更具代表性和多样性的开放领域对话评估基准。我们的方法利用几种最先进的LLM生成用户-聊天机器人多语言对话，这些对话基于多样的种子上下文进行条件生成。然后，一个强大的LLM（GPT-4.1）被用于对聊天机器人的性能进行多维分析，揭示出明显的跨语言性能差异。在指导这种大规模评估的基础上，我们整理了一个新的元评估多语言基准，并对样本进行了人工标注，带有细致的质量判断。该基准随后用于评估几种推理和非推理LLM作为开放领域对话评估者的能力建设。使用MEDAL，我们发现最先进的评判者无法可靠地检测到诸如缺乏同理心、常识或相关性等细微问题。
英文摘要：
Evaluating the quality of open-domain chatbots has become increasingly reliant on LLMs acting as automatic judges. However, existing meta-evaluation benchmarks are static, outdated, and lacking in multilingual coverage, limiting their ability to fully capture subtle weaknesses in evaluation. We introduce MEDAL, an automated multi-agent framework for curating more representative and diverse open-domain dialogue evaluation benchmarks. Our approach leverages several state-of-the-art LLMs to generate user-chatbot multilingual dialogues, conditioned on varied seed contexts. Then, a strong LLM (GPT-4.1) is used for a multidimensional analysis of the performance of the chatbots, uncovering noticeable cross-lingual performance differences. Guided by this large-scale evaluation, we curate a new meta-evaluation multilingual benchmark and human-annotate samples with nuanced quality judgments. This benchmark is then used to assess the ability of several reasoning and non-reasoning LLMs to act as evaluators of open-domain dialogues. Using MEDAL, we uncover that state-of-the-art judges fail to reliably detect nuanced issues such as lack of empathy, commonsense, or relevance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
