MORQA：面向医学开放式问答的评估指标基准测试
MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering
AI总结：
本文提出多语言医学问答基准MORQA，通过包含专家撰写的多参考答案和人工评分，系统评估了传统指标与GPT-4等LLM评估器，发现LLM方法与专家判断的相关性显著更优。
摘要：
在医学领域评估自然语言生成（NLG）系统面临着独特的挑战，因为其对准确性、相关性和领域专业知识有着极高的要求。传统的自动评估指标，如BLEU、ROUGE和BERTScore，往往难以区分高质量的输出，尤其是在医学问答（QA）任务具有开放式性质且可能存在多个有效回答的情况下。在本研究中，我们提出了MORQA（Medical Open-Response QA），这是一个全新的多语言基准，旨在评估NLG评估指标在三个基于英文和中文的医学视觉与文本QA数据集上的有效性。与现有资源不同，我们的数据集包含由医学专业人士撰写的2-4个及以上的黄金标准答案，并附有专家对三个英文和中文子集的人工评分。我们对传统指标和基于大语言模型（LLM）的评估器（如GPT-4和Gemini）进行了基准测试，发现基于LLM的方法在与专家判断的相关性上显著优于传统指标。我们进一步分析了推动这一改进的因素，包括LLM对语义细微差别的敏感度以及对参考答案变异的鲁棒性。我们的结果提供了医学领域NLG评估的首个全面的多语言定性研究，强调了以人为本的评估方法的需求。所有数据集和注释将公开发布以支持未来的研究。
英文摘要：
Evaluating natural language generation (NLG) systems in the medical domain presents unique challenges due to the critical demands for accuracy, relevance, and domain-specific expertise. Traditional automatic evaluation metrics, such as BLEU, ROUGE, and BERTScore, often fall short in distinguishing between high-quality outputs, especially given the open-ended nature of medical question answering (QA) tasks where multiple valid responses may exist. In this work, we introduce MORQA (Medical Open-Response QA), a new multilingual benchmark designed to assess the effectiveness of NLG evaluation metrics across three medical visual and text-based QA datasets in English and Chinese. Unlike prior resources, our datasets feature 2-4+ gold-standard answers authored by medical professionals, along with expert human ratings for three English and Chinese subsets. We benchmark both traditional metrics and large language model (LLM)-based evaluators, such as GPT-4 and Gemini, finding that LLM-based approaches significantly outperform traditional metrics in correlating with expert judgments. We further analyze factors driving this improvement, including LLMs' sensitivity to semantic nuances and robustness to variability among reference answers. Our results provide the first comprehensive, multilingual qualitative study of NLG evaluation in the medical domain, highlighting the need for human-aligned evaluation methods. All datasets and annotations will be publicly released to support future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
