VERT：用于放射学报告评估的可靠LLM评判者
Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies
AI总结：
本文提出VERT作为LLM评判者，通过对比现有指标和实验验证，展示了其在多模态和解剖结构上的鲁棒性及轻量级微调的有效性。
摘要：
当前放射学报告评估文献主要集中在设计基于LLM的度量标准和微调小型模型进行胸部X光分析。然而，这些方法在应用于其他模态和解剖结构的报告时的鲁棒性尚不明确。本文通过专家和LLM评分的相关性分析，比较了三种现有LLM-as-a-judge指标（RadFact、GREEN和FineRadScore）与所提出的VERT指标，使用开源和闭源模型（推理和非推理）在两个专家标注的数据集（RadEval和RaTE-Eval）上进行评估，涵盖多种模态和解剖结构。进一步评估了少样本方法、集成和参数高效微调。通过系统性错误检测和分类研究，评估了这些指标与专家判断的一致性，并识别了一致性和不一致性的区域。结果表明，VERT相比GREEN在与放射科医生判断的相关性上提高了11.7%。此外，仅使用1,300个训练样本微调Qwen3 30B模型，取得了高达25%的提升，微调模型的推理时间减少了37.2倍。这些发现突显了基于LLM的评判者的效果，并证明了通过轻量级适应可以实现可靠的评估。
英文摘要：
Current work on radiology report evaluation has focused primarily on LLM-based metrics for chest X-rays (CXR), leaving open whether such approaches remain robust across other modalities and anatomies, and which model and prompt configurations are best suited to serve as LLM judges. We present a comprehensive evaluation of LLM judges for radiology report evaluation across diverse model families, prompting strategies, and adaptation methods, leveraging two expert-annotated datasets that span multiple imaging modalities and anatomical regions. Specifically, we compare four existing metrics (RadFact, GREEN, FineRadScore, and CRIMSON) with our proposed metric, VERT, across open- and closed-source models of varying sizes and reasoning capabilities. Among existing metrics, GREEN proves the most robust, outperforming more recent CXR-specific approaches. VERT further improves Kendall's correlation with radiologist judgment by up to 26% over GREEN, while ensembling and lightweight fine-tuning of Qwen3-30B yield gains of up to 9.9% and 25%, respectively, of which the latter also has a latency reduction of 37.2x. A systematic error analysis further shows that, while these metrics correlate with expert ratings overall, they tend to miscount errors such as omissions. Overall, our results show that carefully selected general-purpose LLM judges provide strong radiology report evaluation across modalities, with lightweight adaptation offering complementary gains when annotated data is available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
