CTest-Metric：一个统一框架，用于评估CT报告生成中度量的临床有效性
CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation
AI总结：
CTest-Metric通过三个模块评估CT报告生成中度量的临床有效性，发现GREEN Score与专家判断最一致，BERTScore-F1对事实性错误最不敏感。
摘要：
在生成式人工智能时代，即使关键的医疗任务越来越多地被自动化，放射学报告生成（RRG）仍然依赖于次优的度量标准进行质量评估。因此，开发领域特定的度量标准已成为研究的活跃领域，但仍然具有挑战性，因为缺乏一个统一、明确的框架来评估其在临床环境中的稳健性和适用性。为了解决这个问题，我们提出了CTest-Metric，这是一个首个统一的度量评估框架，包含三个模块，用于确定CT RRG中度量的临床可行性。这些模块测试：（i）通过基于LLM的重述进行写作风格泛化性（WSG）；（ii）在不同严重程度下进行合成错误注入（SEI）；以及（iii）使用临床人员对175个“分歧”案例的评分进行度量与专家相关性（MvE）。八个广泛使用的度量标准（BLEU、ROUGE、METEOR、BERTScore-F1、F1-RadGraph、RaTEScore、GREEN Score、CRG）在七个基于CT-CLIP编码器构建的LLM上进行了研究。使用我们新颖的框架，我们发现语言自然生成（NLG）度量标准对风格变化高度敏感；GREEN Score与专家判断最一致（Spearman~0.70），而CRG显示负相关；BERTScore-F1对事实性错误注入最不敏感。我们将发布该框架、代码和匿名评估数据的可接受部分（重述/错误注入的CT报告）以促进可重复的基准测试和未来度量标准的发展。
英文摘要：
In the generative AI era, where even critical medical tasks are increasingly automated, radiology report generation (RRG) continues to rely on suboptimal metrics for quality assessment. Developing domain-specific metrics has therefore been an active area of research, yet it remains challenging due to the lack of a unified, well-defined framework to assess their robustness and applicability in clinical contexts. To address this, we present CTest-Metric, a first unified metric assessment framework with three modules determining the clinical feasibility of metrics for CT RRG. The modules test: (i) Writing Style Generalizability (WSG) via LLM-based rephrasing; (ii) Synthetic Error Injection (SEI) at graded severities; and (iii) Metrics-vs-Expert correlation (MvE) using clinician ratings on 175 "disagreement" cases. Eight widely used metrics (BLEU, ROUGE, METEOR, BERTScore-F1, F1-RadGraph, RaTEScore, GREEN Score, CRG) are studied across seven LLMs built on a CT-CLIP encoder. Using our novel framework, we found that lexical NLG metrics are highly sensitive to stylistic variations; GREEN Score aligns best with expert judgments (Spearman~0.70), while CRG shows negative correlation; and BERTScore-F1 is least sensitive to factual error injection. We will release the framework, code, and allowable portion of the anonymized evaluation data (rephrased/error-injected CT reports), to facilitate reproducible benchmarking and future metric development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
