Clinically-Grounded Counterfactual Reasoning for Medical Video Diagnosis
Center for Data Science in Clinical Medicine（临床医学数据科学中心）；The State Key Lab of Brain-Machine Intelligence（脑机智能国家重点实验室）；Department of Gynecology and Obstetrics, 7th Medical Center of Chinese PLA General Hospital（中国人民解放军第七医学中心妇产科部）；School of Computer Science, Peking University（北京大学计算机学院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理学与认知科学学院）；State Key Lab of General AI, Peking University（通用人工智能国家重点实验室）；Nat’l Eng. Research Center of Visual Technology（视觉技术国家工程研究中心）；Beijing Key Laboratory of Behavior and Mental Health（北京行为与心理健康重点实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，北京大学-武汉人工智能研究院）
Medical video diagnosis involves inferring clinical decisions from dynamic tissue responses throughout examination processes. Existing methods rely on an end-to-end learning paradigm that i) focuses on appearance rather than pathology, ii) lacks clinical priors, and iii) reasons solely from observations without counterfactual comparison. This work introduces MedVCR, a counterfactual reasoning framework that mimics clinical diagnostic thinking. MedVCR comprises three components: a Counterfactual Generator that synthesizes tissue evolution under specified pathological states via a diffusion-based manner; a Counterfactual Representation Learning module that encodes diagnostic knowledge through clinical rules (i.e., temporal consistency, pathological separability, and counterfactual alignment); and a Dual Diagnostic Prediction strategy that integrates video-level assessment with frame-level counterfactual analysis. MedVCR is evaluated under both fully supervised (e.g., colposcopy) and weakly supervised (e.g., colonoscopy) video diagnosis settings, yielding 2.6%-10.2% performance gains compared with leading baselines. Comprehensive ablation studies further validate the effectiveness of each component. The code will be released.
Deep neural networks (DNN) have achieved unprecedented performance in computer-vision tasks almost ubiquitously in business, technology, and science. While substantial efforts are made to engineer highly accurate architectures and provide usable model explanations, most state-of-the-art approaches are first designed for natural vision and then translated to the medical domain. This dissertation seeks to address this gap by proposing novel architectures that integrate the domain-specific constraints of medical imaging into the DNN model and explanation design.
Existing studies on disease diagnostic models focus either on diagnostic model learning for performance improvement or on the visual explanation of a trained diagnostic model. We propose a novel learn-explain-reinforce (LEAR) framework that unifies diagnostic model learning, visual explanation generation (explanation unit), and trained diagnostic model reinforcement (reinforcement unit) guided by the visual explanation. For the visual explanation, we generate a counterfactual map that transforms an input sample to be identified as an intended target label. For example, a counterfactual map can…
COMSATS University Islamabad（伊斯兰堡COMSATS大学）；National Center of Artificial Intelligence（国家人工智能中心）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；Ecole de technologie superieure（高等技术学院）
The image-based diagnosis is now a vital aspect of modern automation assisted diagnosis. To enable models to produce pixel-level diagnosis, pixel-level ground-truth labels are essentially required. However, since it is often not straight forward to obtain the labels in many application domains such as in medical image, classification-based approaches have become the de facto standard to perform the diagnosis. Though they can identify class-salient regions, they may not be useful for diagnosis where capturing all of the evidences is important requirement. Alternatively, a counterfactual explana…
The University of Tokyo（东京大学）；LPIXEL Inc.（LPIXEL公司）；National Institute of Informatics（国立信息学研究所）；University of Cambridge（剑桥大学）；Keio University（庆应义塾大学）
Despite the success of Convolutional Neural Network-based Computer-Aided Diagnosis research, its clinical applications remain challenging. Accordingly, developing medical Artificial Intelligence (AI) fitting into a clinical environment requires identifying/bridging the gap between AI and Healthcare sides. Since the biggest problem in Medical Imaging lies in data paucity, confirming the clinical relevance for diagnosis of research-proven image augmentation techniques is essential. Therefore, we hold a clinically valuable AI-envisioning workshop among Japanese Medical Imaging experts, physicians…
C$^2$LUE: Clinically Aligned Patient-Level Ultrasound Diagnosis via Evidence-Aware Multi-Instance Learning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统全国重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
There are multiple papers published about different AI models for the COVID-19 diagnosis with promising results. Unfortunately according to the reviews many of the papers do not reach the level of sophistication needed for a clinically usable model. In this paper I go through multiple review papers, guidelines, and other relevant material in order to generate more comprehensive requirements for the future papers proposing a AI based diagnosis of the COVID-19 from chest X-ray data (CXR). Main findings are that a clinically usable AI needs to have an extremely good documentation, comprehensive s…
See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis
Electrical and Computer Engineering Department, The University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）；Vector Institute（向量研究所）；ETH Zurich（苏黎世联邦理工学院）；Renmin University of China（中国人民大学）；National University of Singapore（新加坡国立大学）
University of Maryland School of Medicine（马里兰大学医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Chest X-ray (CXR) datasets hosted on Kaggle, though useful from a data science competition standpoint, have limited utility in clinical use because of their narrow focus on diagnosing one specific disease. In real-world clinical use, multiple diseases need to be considered since they can co-exist in the same patient. In this work, we demonstrate how federated learning (FL) can be used to make these toy CXR datasets from Kaggle clinically useful. Specifically, we train a single FL classification model (`global`) using two separate CXR datasets -- one annotated for presence of pneumonia and the …
Melanoma is a leading cause of deaths due to skin cancer deaths and hence, early and effective diagnosis of melanoma is of interest. Current approaches for automated diagnosis of melanoma either use pattern recognition or analytical recognition like ABCDE (asymmetry, border, color, diameter and evolving) criterion. In practice however, a differential approach wherein outliers (ugly duckling) are detected and used to evaluate nevi/lesions. Incorporation of differential recognition in Computer Aided Diagnosis (CAD) systems has not been explored but can be beneficial as it can provide a clinical …
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（伊萨尔河畔临床教学中心）
Deep learning models used in medical image analysis are prone to raising reliability concerns due to their black-box nature. To shed light on these black-box models, previous works predominantly focus on identifying the contribution of input features to the diagnosis, i.e., feature attribution. In this work, we explore counterfactual explanations to identify what patterns the models rely on for diagnosis. Specifically, we investigate the effect of changing features within chest X-rays on the classifier's output to understand its decision mechanism. We leverage a StyleGAN-based approach (StyleE…
In this paper, our focus is on constructing models to assist a clinician in the diagnosis of COVID-19 patients in situations where it is easier and cheaper to obtain X-ray data than to obtain high-quality images like those from CT scans. Deep neural networks have repeatedly been shown to be capable of constructing highly predictive models for disease detection directly from image data. However, their use in assisting clinicians has repeatedly hit a stumbling block due to their black-box nature. Some of this difficulty can be alleviated if predictions were accompanied by explanations expressed …
Clinically, the accurate annotation of lesions/tissues can significantly facilitate the disease diagnosis. For example, the segmentation of optic disc/cup (OD/OC) on fundus image would facilitate the glaucoma diagnosis, the segmentation of skin lesions on dermoscopic images is helpful to the melanoma diagnosis, etc. With the advancement of deep learning techniques, a wide range of methods proved the lesions/tissues segmentation can also facilitate the automated disease diagnosis models. However, existing methods are limited in the sense that they can only capture static regional correlations i…
University of Washington（华盛顿大学）；University of Vermont College of Medicine（佛蒙特大学医学院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Training end-to-end networks for classifying gigapixel size histopathological images is computationally intractable. Most approaches are patch-based and first learn local representations (patch-wise) before combining these local representations to produce image-level decisions. However, dividing large tissue structures into patches limits the context available to these networks, which may reduce their ability to learn representations from clinically relevant structures. In this paper, we introduce a novel attention-based network, the Holistic ATtention Network (HATNet) to classify breast biops…
Clinically-guided Data Synthesis for Laryngeal Lesion Detection
Biomedical Robotics Laboratory, Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy Department of Informatics, Bioengineering, Robotics, Systems Engineering, University of Genova, Genoa, Italy Departament de Matem\`atiques i Inform\`atica, Universitat de Barcelona, Spain Computer Vision Center, Bellaterra, Spain Instituci\`o Catalana de Recerca i Estudis Avan cats (ICREA), Barcelona, Spain
TCM-Tongue: A Standardized Tongue Image Dataset with Pathological Annotations for AI-Assisted TCM Diagnosis
Beijing Technology and Business University（北京工商大学）；Xiyuan Hospital of China Academy of Chinese Medical Sciences（中国中医科学院西苑医院）；Peking University People's Hospital（北京大学人民医院）
Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding
University of California, Los Angeles（加州大学洛杉矶分校）；University of Pittsburgh（匹兹堡大学）；Fudan University（复旦大学）；University of California, Riverside（加州大学河滨分校）；Hong Kong University of Science（香港科学大学）；Maharishi International University（玛希拉国际大学）
INESC-ID（里斯本系统与计算机工程研究所）；Instituto Superior Técnico, University of Lisbon（里斯本大学高等技术学院）；Lisbon School of Medicine, University of Lisbon（里斯本大学里斯本医学院）
Coronary X-ray angiography is a crucial clinical procedure for the diagnosis and treatment of coronary artery disease, which accounts for roughly 16% of global deaths every year. However, the images acquired in these procedures have low resolution and poor contrast, making lesion detection and assessment challenging. Accurate coronary artery segmentation not only helps mitigate these problems, but also allows the extraction of relevant anatomical features for further analysis by quantitative methods. Although automated segmentation of coronary arteries has been proposed before, previous approa…
MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Center for Brain-Inspired Computing Research (CBICR), Department of Precision Instrument, Tsinghua University（脑启发计算研究中心（CBICR）、精密仪器系，清华大学）
CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale
The Key Laboratory of Smart Human-Computer Interaction（智能人机交互关键实验室）；Wearable Technology of Shaanxi Province, Xidian University, Xi'an, China（陕西 wearable 技术, 西安电子科技大学, 中国）；Nanjing University of Science and Technology（南京理工大学）
Bridging Brain Connectomes and Clinical Reports for Early Alzheimer's Disease Diagnosis
The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校）；University of Cambridge（剑桥大学）；University of Sheffield（谢菲尔德大学）；The University of Georgia（佐治亚大学）
Realism in Action: Anomaly-Aware Diagnosis of Brain Tumors from Medical Images Using YOLOv8 and DeiT
Department of CS \& IT Institute for Advanced Studies in Basic Sciences (IASBS) Zanjan, Iran；Department of Computer Engineering University of Zanjan (ZNU) Zanjan, Iran
Towards Clinically Interpretable Ophthalmic VQA via Spatially-Grounded Lesion Evidence
Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；The Hong Kong Polytechnic University（香港理工大学）；National University of Singapore（新加坡国立大学）；University of Washington（华盛顿大学）；Institute of High Performance Computing, Agency for Science, Technology and Research（科技研究局高性能计算研究所）
New York University（纽约大学）；NYU Langone Health（纽约大学朗格尼医学中心）；Warsaw University of Technology（华沙理工大学）
Deep neural networks (DNNs) show promise in image-based medical diagnosis, but cannot be fully trusted since their performance can be severely degraded by dataset shifts to which human perception remains invariant. If we can better understand the differences between human and machine perception, we can potentially characterize and mitigate this effect. We therefore propose a framework for comparing human and machine perception in medical diagnosis. The two are compared with respect to their sensitivity to the removal of clinically meaningful information, and to the regions of an image deemed m…
Revolutionizing Disease Diagnosis with simultaneous functional PET/MR and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks
School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Zhongshan Hospital（中山医院）；State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University（上海科技大学先进医用材料与器件国家重点实验室）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
Mitigating Overfitting in Medical Imaging: Self-Supervised Pretraining vs. ImageNet Transfer Learning for Dermatological Diagnosis
Dpt. of Signal Theory and Communications University of Seville（信号理论与通信系塞维利亚大学）；Department of Dermatology Hospital Universitario Virgen Macarena（皮肤科部门塞维利亚大学医院）；Department of Dermatology Hospital Quirón Salud Sevilla（皮肤科部门Quirón Salud Sevilla医院）
