Hallucination Filtering in Radiology Vision-Language Models Using Discrete Semantic Entropy
Lab for Artificial Intelligence in Medicine, Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(医学人工智能实验室,诊断与介入放射学部,RWTH亚琛大学医院);Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(诊断与介入放射学部,RWTH亚琛大学医院);Department of Diagnostic and Interventional Radiology, Technical University of Munich, School of Medicine and Health, Klinikum rechts der Isar, TUM University Hospital(诊断与介入放射学部,慕尼黑技术大学,医学院与健康学院,Klinikum rechts der Isar,TUM大学医院);Department of Cardiovascular Radiology and Nuclear Medicine, Technical University of Munich, School of Medicine and Health, German Heart Center, TUM University Hospital(心血管放射学与核医学部,慕尼黑技术大学,医学院与健康学院,德国心脏中心,TUM大学医院);Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology(数字健康中心,医学院与卡尔·古斯塔夫·卡尔斯大学医院,德累斯顿技术大学);Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology(第一医学部,医学院与卡尔·古斯塔夫·卡尔斯大学医院,德累斯顿技术大学);Pathology & Data Analytics, Leeds Institute of Medical Research at St James’s, University of Leeds(病理学与数据分析,圣詹姆斯医院医学研究所,利兹大学);Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg(医学肿瘤学,国家肿瘤疾病中心(NCT),海德堡大学医院)
Patrick Wienholt, Sophie Caselitz, Robert Siepmann, Philipp Bruners, Keno Bressem, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, Daniel Truhn
To determine whether using discrete semantic entropy (DSE) to reject questions likely to generate hallucinations can improve the accuracy of black-box vision-language models (VLMs) in radiologic image based visual question answering (VQA). This retrospective study evaluated DSE using two publicly available, de-identified datasets: the VQA-Med 2019 benchmark (500 images with clinical questions and short-text answers) and a diagnostic radiology dataset (206 cases: 60 computed tomography scans, 60 magnetic resonance images, 60 radiographs, 26 angiograms) with corresponding ground-truth diagnoses. GPT-4o and GPT-4.1 (Generative Pretrained Transformer; OpenAI) answered each question 15 times using a temperature of 1.0. Baseline accuracy was determined using low-temperature answers (temperature 0.1). Meaning-equivalent responses were grouped using bidirectional entailment checks, and DSE was computed from the relative frequencies of the resulting semantic clusters. Accuracy was recalculated after excluding questions with DSE > 0.6 or > 0.3. p-values and 95% confidence intervals were obtained using bootstrap resampling and a Bonferroni-corrected threshold of p < .004 for statistical significance. Across 706 image-question pairs, baseline accuracy was 51.7% for GPT-4o and 54.8% for GPT-4.1. After filtering out high-entropy questions (DSE > 0.3), accuracy on the remaining questions was 76.3% (retained questions: 334/706) for GPT-4o and 63.8% (retained questions: 499/706) for GPT-4.1 (both p < .001). Accuracy gains were observed across both datasets and largely remained statistically significant after Bonferroni correction. DSE enables reliable hallucination detection in black-box VLMs by quantifying semantic inconsistency. This method significantly improves diagnostic answer accuracy and offers a filtering strategy for clinical VLM applications.
Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
Department of computer science, University of Central Florida, Orlando, USA(计算机科学系,中央佛罗里达大学);Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA(生物工程系,宾夕法尼亚大学);Department of electrical engineering, Columbia university, New York, NY, USA(电气工程系,哥伦比亚大学);Technical University of Applied Sciences Regensburg, Regensburg, Germany(应用科学技术大学(雷根斯堡));Department of Surgery, University of Calgary, Calgary, Alberta, Canada(外科系,卡尔加里大学);University College of Nabi Akram, Tabriz, Iran(纳比阿克兰大学);School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran(电气工程学院,伊朗科学技术大学)
Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering
University of New South Wales(新南威尔士大学);Australian Artificial Intelligence Institute, University of Technology Sydney(澳大利亚人工智能研究所,技术悉尼大学);Stanford University(斯坦福大学);School of Computing and Information Technology of University of Wollongong Australia(沃林根澳大利亚大学计算与信息科技学院);Sun Yat-sen University(中山大学);University of Science and Technology of China(中国科学技术大学)
Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence
Huazhong University of Science and Technology(华中科技大学);Imperial Global Singapore, Imperial College London(帝国理工学院新加坡全球中心);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Anhui University(安徽大学)
VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China(复旦大学计算机科学与人工智能学院,上海,中国);Shanghai Key Laboratory of Intelligent Information Processing(上海智能信息处理重点实验室)
Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation
Harvard University(哈佛大学);Jawaharlal Institute of Postgraduate Medical Education & Research(贾瓦哈拉尔 postgraduate 医学教育与研究学院);Johns Hopkins University(约翰斯·霍普金斯大学)
BTReport: A Framework for Brain Tumor Radiology Report Generation with Clinically Relevant Features
University of Washington(华盛顿大学);University of Washington School of Medicine(华盛顿大学医学院);Microsoft Health AI(微软健康人工智能);Johns Hopkins School of Medicine(约翰霍普金斯医学院)
A Utility-preserving De-identification Pipeline for Cross-hospital Radiology Data Sharing
Shandong University(山东大学);Australian National University(澳大利亚国立大学);Hong Kong University of Science and Technology(香港科技大学);Curtin University(科廷大学);Edith Cowan University(埃迪斯科文大学);University of Western Australia(西澳大利亚大学)
Towards Statistical Factuality Guarantee for Large Vision-Language Models
Vanderbilt University(范德堡大学);Vanderbilt University Medical Center(范德堡大学医学中心);Intuit(英图伊特公司);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Intuit AI Research(英图伊特人工智能研究院)
Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);College of AI, Tsinghua University(清华大学人工智能学院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Beijing Electronic Digital & Intelligence(北京电子数字与智能)
Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models
University of Zurich and University Hospital of Zurich(苏黎世大学及苏黎世大学医院);Kobe University(Kobe大学);ETH AI Center(苏黎世联邦理工学院人工智能中心);ETH Zurich(苏黎世联邦理工学院);Stanford University(斯坦福大学);Zurich University of Applied Sciences(苏黎世应用科学大学)
Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports
Rensselaer Polytechnic Institute, NY, USA(罗文学院);IBM Research, Almaden, CA, USA(IBM研究院);Stanford University, CA, USA(斯坦福大学);Massachusetts General Hospital (MGH), Boston, USA(麻省总医院)
Halt the Hallucination: Decoupling Signal and Semantic OOD Detection Based on Cascaded Early Rejection
School of Computer and Electronic Information, Nanjing Normal University, China(南京师范大学计算机与电子信息学院);School of Artificial Intelligence and Information Technology, Nanjing University of Chinese Medicine, China(南京中医药大学人工智能与信息技术学院)