Current approaches to explaining the decisions of deep learning systems for medical tasks have focused on visualising the elements that have contributed to each decision. We argue that such approaches are not enough to "open the black box" of medical decision making systems because they are missing a key component that has been used as a standard communication tool between doctors for centuries: language. We propose a model-agnostic interpretability method that involves training a simple recurrent neural network model to produce descriptive sentences to clarify the decision of deep learning classifiers.
We test our method on the task of detecting hip fractures from frontal pelvic x-rays. This process requires minimal additional labelling despite producing text containing elements that the original deep learning classification model was not specifically trained to detect.
The experimental results show that: 1) the sentences produced by our method consistently contain the desired information, 2) the generated sentences are preferred by doctors compared to current tools that create saliency maps, and 3) the combination of visualisations and generated text is better than either alone.
Knowledge-driven AI-generated data for accurate and interpretable breast ultrasound diagnoses
Peking University(北京大学);Peking University Cancer Hospital & Institute(北京大学肿瘤医院);Peking Union Medical College Hospital(北京协和医院);Cancer Hospital, Chinese Academy of Medical Sciences(中国医学科学院肿瘤医院);Yizhun Medical AI Co., Ltd(医准智能医疗科技有限公司);Nanchang People’s Hospital(南昌市人民医院)
FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation
University of Arkansas(阿肯色大学);West Virginia University(西弗吉尼亚大学);University of Liverpool(利物浦大学);MD Anderson Cancer Center(MD安德森癌症中心);University of Houston(休斯顿大学)
Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk Categorization
New York University(纽约大学);Columbia University Irving Medical Center(哥伦比亚大学欧文医学中心);NYU Grossman School of Medicine(纽约大学格罗斯曼医学院);University of California San Diego(加州大学圣地亚哥分校)
Interpretable Medical Image Classification using Prototype Learning and Privileged Information
Experimental Radiology, University Hospital Ulm, Germany(实验放射学,乌尔姆大学医院,德国);Department of Diagnostic and Interventional Radiology, University Hospital Ulm, Germany(诊断与介入放射学系,乌尔姆大学医院,德国);i2SouI - Innovative Imaging in Surgical Oncology Ulm, University Hospital Ulm, Germany(i2SouI - 精准影像学在手术肿瘤学中的创新应用,乌尔姆大学医院,德国)
MedForge: Interpretable Medical Deepfake Detection via Forgery-aware Reasoning
National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Hunan University(湖南大学);Xi’an Jiaotong University(西安交通大学);Guangdong Provincial People’s Hospital(广东省人民医院)
RadGame: An AI-Powered Platform for Radiology Education
Harvard Medical School(哈佛医学院);Mass General Brigham(麻省总医院);Maastricht University(马斯特里赫特大学);Department of Medical Imaging, King Abdulaziz Medical City, Ministry of National Guard, Riyadh, Saudi Arabia(国王阿卜杜勒-阿齐兹医疗城医学影像科,沙特阿拉伯);National Strategic Technology Research Institute, Seoul National University Hospital(全国战略技术研究所,首尔国立大学医院);Saint Louis University School of Medicine(圣路易斯大学医学院);College of Medicine, King Saud bin Abdulaziz University for Health Sciences(国王萨勒曼·本·阿卜杜勒阿齐兹大学医学院);Tufts University School of Medicine(塔夫茨大学医学院);Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系)