arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.09256cs.CV
收藏

在放射学视觉-语言模型中使用离散语义熵过滤幻觉

Hallucination Filtering in Radiology Vision-Language Models Using Discrete Semantic Entropy

Lab for Artificial Intelligence in Medicine, Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(医学人工智能实验室,诊断与介入放射学部,RWTH亚琛大学医院);Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(诊断与介入放射学部,RWTH亚琛大学医院);Department of Diagnostic and Interventional Radiology, Technical University of Munich, School of Medicine and Health, Klinikum rechts der Isar, TUM University Hospital(诊断与介入放射学部,慕尼黑技术大学,医学院与健康学院,Klinikum rechts der Isar,TUM大学医院);Department of Cardiovascular Radiology and Nuclear Medicine, Technical University of Munich, School of Medicine and Health, German Heart Center, TUM University Hospital(心血管放射学与核医学部,慕尼黑技术大学,医学院与健康学院,德国心脏中心,TUM大学医院);Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology(数字健康中心,医学院与卡尔·古斯塔夫·卡尔斯大学医院,德累斯顿技术大学);Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology(第一医学部,医学院与卡尔·古斯塔夫·卡尔斯大学医院,德累斯顿技术大学);Pathology & Data Analytics, Leeds Institute of Medical Research at St James’s, University of Leeds(病理学与数据分析,圣詹姆斯医院医学研究所,利兹大学);Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg(医学肿瘤学,国家肿瘤疾病中心(NCT),海德堡大学医院)

Patrick Wienholt, Sophie Caselitz, Robert Siepmann, Philipp Bruners, Keno Bressem, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, Daniel Truhn

AI总结:

本研究通过离散语义熵过滤高熵问题,显著提升放射学视觉-语言模型的诊断准确性。

摘要:

为确定使用离散语义熵(DSE)来拒绝可能生成幻觉的问题是否能提高黑箱视觉-语言模型(VLMs)在基于放射图像的视觉问答(VQA)中的准确性,本回顾性研究评估了DSE。研究使用了两个公开可用的去标识化数据集:VQA-Med 2019基准(500张带有临床问题和短文本答案的图像)和一个诊断放射学数据集(206例:60例CT扫描、60例磁共振成像、60例X光片、26例血管造影图像),并附有相应的 ground-truth 诊断。GPT-4o 和 GPT-4.1(生成式预训练变换器;OpenAI)使用温度为1.0的答案每个问题回答15次。基线准确性通过低温答案(温度0.1)确定。通过双向蕴含检查将意义等价响应分组,并从所得到的语义簇的相对频率计算DSE。在排除DSE>0.6或>0.3的问题后重新计算准确性。通过bootstrap重采样和Bonferroni校正阈值p<.004获得p值和95%置信区间。在706张图像-问题对中,基线准确性为GPT-4o 51.7%,GPT-4.1 54.8%。在过滤掉高熵问题(DSE>0.3)后,剩余问题的准确性为GPT-4o 76.3%(保留问题:334/706),GPT-4.1 63.8%(保留问题:499/706)(两者p<.001)。在两个数据集中均观察到准确性提升,并且在Bonferroni校正后仍保持统计学显著性。DSE通过量化语义不一致性在黑箱VLMs中实现可靠的幻觉检测。该方法显著提高了诊断答案的准确性,并为临床VLM应用提供了过滤策略。

英文摘要:

To determine whether using discrete semantic entropy (DSE) to reject questions likely to generate hallucinations can improve the accuracy of black-box vision-language models (VLMs) in radiologic image based visual question answering (VQA). This retrospective study evaluated DSE using two publicly available, de-identified datasets: the VQA-Med 2019 benchmark (500 images with clinical questions and short-text answers) and a diagnostic radiology dataset (206 cases: 60 computed tomography scans, 60 magnetic resonance images, 60 radiographs, 26 angiograms) with corresponding ground-truth diagnoses. GPT-4o and GPT-4.1 (Generative Pretrained Transformer; OpenAI) answered each question 15 times using a temperature of 1.0. Baseline accuracy was determined using low-temperature answers (temperature 0.1). Meaning-equivalent responses were grouped using bidirectional entailment checks, and DSE was computed from the relative frequencies of the resulting semantic clusters. Accuracy was recalculated after excluding questions with DSE > 0.6 or > 0.3. p-values and 95% confidence intervals were obtained using bootstrap resampling and a Bonferroni-corrected threshold of p < .004 for statistical significance. Across 706 image-question pairs, baseline accuracy was 51.7% for GPT-4o and 54.8% for GPT-4.1. After filtering out high-entropy questions (DSE > 0.3), accuracy on the remaining questions was 76.3% (retained questions: 334/706) for GPT-4o and 63.8% (retained questions: 499/706) for GPT-4.1 (both p < .001). Accuracy gains were observed across both datasets and largely remained statistically significant after Bonferroni correction. DSE enables reliable hallucination detection in black-box VLMs by quantifying semantic inconsistency. This method significantly improves diagnostic answer accuracy and offers a filtering strategy for clinical VLM applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑