用于医疗对话系统可靠评估的自动化评分标准
Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluation
AI总结：
本文提出一种检索增强的多智能体框架，用于自动生成实例特定的评估评分标准，通过权威医学证据生成可验证的细粒度评估标准，并在HealthBench和LLMEval-Med数据集上取得显著成效。
摘要：
大型语言模型（LLMs）在临床决策支持中的应用日益增加，但幻觉和不安全建议可能直接威胁患者安全。这些风险难以评估：细微的临床错误常被通用指标和LLM判官根据通用标准所遗漏，而由专家撰写的细粒度评分标准成本高且难以扩展。本文提出一种检索增强的多智能体框架，用于自动化生成实例特定的评估评分标准。我们的方法通过将检索内容分解为原子事实，并与用户交互约束相结合，形成可验证的细粒度评估标准。在HealthBench和LLMEval-Med数据集上评估，我们的框架实现了临床意图对齐（CIA）得分为50.20%和31.90%，显著优于GPT-4o基线，并展示了鲁棒的跨语言泛化能力。在HealthBench的判别测试中，我们的评分标准比GPT-4o基线高出7.8%的胜率，且得分差异$Δ$几乎翻倍。消融研究确认了其结构必要性。除了评估，我们的评分标准还有效指导响应优化，提高了9.2%的质量。这为评估和改进医疗LLMs提供了可扩展的跨语言基础。代码可在https://github.com/AmbeChen/Automated-Rubric-Generation获取。
英文摘要：
Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety. These risks are hard to assess: subtle clinical errors are often missed by generic metrics and LLM judges using general criteria, while expert-authored fine-grained rubrics are expensive and difficult to scale. In this paper, we propose a retrieval-augmented multi-agent framework for automatically generating instance-specific evaluation rubrics. Our approach grounds evaluation in authoritative medical evidence by decomposing retrieved content into atomic facts and synthesizing them with user interaction constraints to form fine-grained evaluation criteria. Evaluated on HealthBench and LLMEval-Med, our framework achieves Clinical Intent Alignment (CIA) scores of 50.20% and 31.90%, significantly outperforming the GPT-4o baseline and showing consistent improvements across English and Chinese medical benchmarks. In discriminative tests on HealthBench, our rubrics achieve a 7.8% point higher win rate than GPT-4o and increase the mean score difference from 4.972 to 8.658. Ablation studies further show that individual components contribute differently across datasets, with interaction-intent modeling providing the most consistent contribution to clinical-criterion coverage. Beyond evaluation, our rubrics guide response refinement, improving response quality by 9.2%. These results suggest that automated, knowledge-grounded rubric generation provides a scalable foundation for evaluating and improving medical LLMs. The code is available at https://anonymous.4open.science/r/Automated-Rubric-Generation-E716/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
