多RADS合成放射学报告数据集及41种开源和专有语言模型的头对头基准测试
Multi-RADS Synthetic Radiology Report Dataset and Head-to-Head Benchmarking of 41 Open-Weight and Proprietary Language Models
AI总结:
本研究提出了多RADS合成放射学报告数据集RXL-RADSet,并比较了41种开源和专有语言模型在RADS分配任务中的性能。
摘要:
背景:报告和数据系统(RADS)标准化放射学风险沟通,但从叙述报告中自动分配RADS极具挑战性,因为指南复杂性、输出格式限制以及跨RADS框架和模型大小的有限基准测试。目的:创建RXL-RADSet,一个由放射科医生验证的多RADS基准,并比较开源小型语言模型(SLMs)与专有模型在RADS分配中的有效性和准确性。材料和方法:RXL-RADSet包含1,600个合成放射学报告,涵盖10种RADS(BI-RADS、CAD-RADS、GB-RADS、LI-RADS、Lung-RADS、NI-RADS、O-RADS、PI-RADS、TI-RADS、VI-RADS)和多种成像模态。报告由LLMs根据场景计划和模拟放射科医生风格生成,并经过两阶段放射科医生验证。我们评估了41种量化SLMs(12种家族,0.135-32B参数)和GPT-5.2,在固定引导提示下。主要终点是有效性和准确性;次要分析比较引导提示与零shot提示。结果:在引导提示下,GPT-5.2实现了99.8%的有效性和81.1%的准确性(1,600次预测)。汇总SLMs(65,600次预测)实现了96.8%的有效性和61.1%的准确性;20-32B范围内的顶级SLMs实现了约99%的有效性和中等到较高的70%准确性。性能随着模型大小增加(在<1B和>=10B之间出现拐点)而增加,随着RADS复杂性增加而下降,主要由于分类难度而非无效输出。引导提示提高了有效性(99.2% vs 96.7%)和准确性(78.5% vs 69.6%)与零shot提示相比。结论:RXL-RADSet提供了一个由放射科医生验证的多RADS基准;大SLMs(20-32B)在引导提示下可以接近专有模型的性能,但更高复杂度的方案仍有差距。
英文摘要:
Background: Reporting and Data Systems (RADS) standardize radiology risk communication but automated RADS assignment from narrative reports is challenging because of guideline complexity, output-format constraints, and limited benchmarking across RADS frameworks and model sizes. Purpose: To create RXL-RADSet, a radiologist-verified synthetic multi-RADS benchmark, and compare validity and accuracy of open-weight small language models (SLMs) with a proprietary model for RADS assignment. Materials and Methods: RXL-RADSet contains 1,600 synthetic radiology reports across 10 RADS (BI-RADS, CAD-RADS, GB-RADS, LI-RADS, Lung-RADS, NI-RADS, O-RADS, PI-RADS, TI-RADS, VI-RADS) and multiple modalities. Reports were generated by LLMs using scenario plans and simulated radiologist styles and underwent two-stage radiologist verification. We evaluated 41 quantized SLMs (12 families, 0.135-32B parameters) and GPT-5.2 under a fixed guided prompt. Primary endpoints were validity and accuracy; a secondary analysis compared guided versus zero-shot prompting. Results: Under guided prompting GPT-5.2 achieved 99.8% validity and 81.1% accuracy (1,600 predictions). Pooled SLMs (65,600 predictions) achieved 96.8% validity and 61.1% accuracy; top SLMs in the 20-32B range reached ~99% validity and mid-to-high 70% accuracy. Performance scaled with model size (inflection between <1B and >=10B) and declined with RADS complexity primarily due to classification difficulty rather than invalid outputs. Guided prompting improved validity (99.2% vs 96.7%) and accuracy (78.5% vs 69.6%) compared with zero-shot. Conclusion: RXL-RADSet provides a radiologist-verified multi-RADS benchmark; large SLMs (20-32B) can approach proprietary-model performance under guided prompting, but gaps remain for higher-complexity schemes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。