谁评判法官？评估LLM作为法官在法语医学开放性问答中的表现
Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
AI总结：
本研究评估了LLM在法语医学开放性问答中作为评判者的有效性，发现领域适应模型在与专家注释一致方面表现最佳，并通过微调和GRPO提升了性能。
摘要：
自动评估医学开放性问答（OEQA）仍具有挑战性，因为需要专家注释。我们评估大型语言模型（LLMs）是否可以作为法语医学OEQA中语义等价性的评判者，比较封闭访问、通用型和生物医学领域适应的模型。我们的结果表明，基于LLM的评判受到生成答案的模型有很大影响，不同生成器之间的同意程度差异显著。领域适应和大型通用模型在与专家注释的一致性方面表现最佳。我们进一步表明，通过监督微调（SFT）和群体相对策略优化（GRPO）对紧凑模型进行轻量级适应，即使在数据有限的情况下，也能显著提高性能并减少生成器敏感性。总体而言，我们的发现强调了生成器意识评估的必要性，并表明经过精心适应的小型模型可以支持低资源医学环境中的可扩展评估。
英文摘要：
Automatic evaluation of medical open-ended question answering (OEQA) remains challenging due to the need for expert annotations. We evaluate whether large language models (LLMs) can act as judges of semantic equivalence in French medical OEQA, comparing closed-access, general-purpose, and biomedical domain-adapted models. Our results show that LLM-based judgments are strongly influenced by the model that generated the answer, with agreement varying substantially across generators. Domain-adapted and large general-purpose models achieve the highest alignment with expert annotations. We further show that lightweight adaptation of a compact model using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) substantially improves performance and reduces generator sensitivity, even with limited data. Overall, our findings highlight the need for generator-aware evaluation and suggest that carefully adapted small models can support scalable evaluation in low-resource medical settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
