arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.03232cs.CLcs.AI
收藏

多RADS合成放射学报告数据集及41种开源和专有语言模型的头对头基准测试

Multi-RADS Synthetic Radiology Report Dataset and Head-to-Head Benchmarking of 41 Open-Weight and Proprietary Language Models

Postgraduate Institute of Medical Education and Research(医学教育与研究研究生院);Tata Medical Center(塔塔医学中心);All India Institute of Medical Sciences(全印度医学科学研究所);National Cancer Institute(国家癌症研究所);Banaras Hindu University(巴纳尔斯·胡斯·印度大学);Aster Malabar Institute of Medical Sciences(Aster马拉巴医学科学研究所)
Kartik Bose, Abhinandan Kumar, Raghuraman Soundararajan, Priya Mudgil, Samonee Ralmilay, Niharika Dutta, Manphool Singhal, Arun Kumar, Saugata Sen, Anurima Patr… 展开作者

Kartik Bose, Abhinandan Kumar, Raghuraman Soundararajan, Priya Mudgil, Samonee Ralmilay, Niharika Dutta, Manphool Singhal, Arun Kumar, Saugata Sen, Anurima Patra, Priya Ghosh, Abanti Das, Amit Gupta, Ashish Verma, Dipin Sudhakaran, Ekta Dhamija, Himangi Unde, Ishan Kumar, Krithika Rangarajan, Prerna Garg, Rachel Sequeira, Sudhin Shylendran, Taruna Yadav, Tej Pal, Pankaj Gupta

AI总结:

本研究提出了多RADS合成放射学报告数据集RXL-RADSet,并比较了41种开源和专有语言模型在RADS分配任务中的性能。

摘要:

背景:报告和数据系统(RADS)标准化放射学风险沟通,但从叙述报告中自动分配RADS极具挑战性,因为指南复杂性、输出格式限制以及跨RADS框架和模型大小的有限基准测试。目的:创建RXL-RADSet,一个由放射科医生验证的多RADS基准,并比较开源小型语言模型(SLMs)与专有模型在RADS分配中的有效性和准确性。材料和方法:RXL-RADSet包含1,600个合成放射学报告,涵盖10种RADS(BI-RADS、CAD-RADS、GB-RADS、LI-RADS、Lung-RADS、NI-RADS、O-RADS、PI-RADS、TI-RADS、VI-RADS)和多种成像模态。报告由LLMs根据场景计划和模拟放射科医生风格生成,并经过两阶段放射科医生验证。我们评估了41种量化SLMs(12种家族,0.135-32B参数)和GPT-5.2,在固定引导提示下。主要终点是有效性和准确性;次要分析比较引导提示与零shot提示。结果:在引导提示下,GPT-5.2实现了99.8%的有效性和81.1%的准确性(1,600次预测)。汇总SLMs(65,600次预测)实现了96.8%的有效性和61.1%的准确性;20-32B范围内的顶级SLMs实现了约99%的有效性和中等到较高的70%准确性。性能随着模型大小增加(在<1B和>=10B之间出现拐点)而增加,随着RADS复杂性增加而下降,主要由于分类难度而非无效输出。引导提示提高了有效性(99.2% vs 96.7%)和准确性(78.5% vs 69.6%)与零shot提示相比。结论:RXL-RADSet提供了一个由放射科医生验证的多RADS基准;大SLMs(20-32B)在引导提示下可以接近专有模型的性能,但更高复杂度的方案仍有差距。

英文摘要:

Background: Reporting and Data Systems (RADS) standardize radiology risk communication but automated RADS assignment from narrative reports is challenging because of guideline complexity, output-format constraints, and limited benchmarking across RADS frameworks and model sizes. Purpose: To create RXL-RADSet, a radiologist-verified synthetic multi-RADS benchmark, and compare validity and accuracy of open-weight small language models (SLMs) with a proprietary model for RADS assignment. Materials and Methods: RXL-RADSet contains 1,600 synthetic radiology reports across 10 RADS (BI-RADS, CAD-RADS, GB-RADS, LI-RADS, Lung-RADS, NI-RADS, O-RADS, PI-RADS, TI-RADS, VI-RADS) and multiple modalities. Reports were generated by LLMs using scenario plans and simulated radiologist styles and underwent two-stage radiologist verification. We evaluated 41 quantized SLMs (12 families, 0.135-32B parameters) and GPT-5.2 under a fixed guided prompt. Primary endpoints were validity and accuracy; a secondary analysis compared guided versus zero-shot prompting. Results: Under guided prompting GPT-5.2 achieved 99.8% validity and 81.1% accuracy (1,600 predictions). Pooled SLMs (65,600 predictions) achieved 96.8% validity and 61.1% accuracy; top SLMs in the 20-32B range reached ~99% validity and mid-to-high 70% accuracy. Performance scaled with model size (inflection between <1B and >=10B) and declined with RADS complexity primarily due to classification difficulty rather than invalid outputs. Guided prompting improved validity (99.2% vs 96.7%) and accuracy (78.5% vs 69.6%) compared with zero-shot. Conclusion: RXL-RADSet provides a radiologist-verified multi-RADS benchmark; large SLMs (20-32B) can approach proprietary-model performance under guided prompting, but gaps remain for higher-complexity schemes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑