Retinal image matching plays a crucial role in monitoring disease progression and treatment response. However, datasets with matched keypoints between temporally separated pairs of images are not available in abundance to train transformer-based model. We propose a novel approach based on reverse knowledge distillation to train large models with limited data while preventing overfitting. Firstly, we propose architectural modifications to a CNN-based semi-supervised method called SuperRetina that help us improve its results on a publicly available dataset. Then, we train a computationally heavier model based on a vision transformer encoder using the lighter CNN-based model, which is counter-intuitive in the field knowledge-distillation research where training lighter models based on heavier ones is the norm. Surprisingly, such reverse knowledge distillation improves generalization even further. Our experiments suggest that high-dimensional fitting in representation space may prevent overfitting unlike training directly to match the final output. We also provide a public dataset with annotations for retinal image keypoint detection and matching to help the research community develop algorithms for retinal image applications.
Uncertainty-Aware Retinal Vessel Segmentation via Ensemble Distillation
University of Ibadan（伊巴迪安大学）；African Institute for Mathematical Sciences（非洲数学科学研究所）；University College London（伦敦大学学院）；Hertie Institute for Brain Health（海德研究所）
Monash University（莫纳什大学）；Airdoc；Kyushu Institute of Technology（九州工业大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）
The need for comprehensive and automated screening methods for retinal image classification has long been recognized. Well-qualified doctors annotated images are very expensive and only a limited amount of data is available for various retinal diseases such as age-related macular degeneration (AMD) and diabetic retinopathy (DR). Some studies show that AMD and DR share some common features like hemorrhagic points and exudation but most classification algorithms only train those disease models independently. Inspired by knowledge distillation where additional monitoring signals from various sour…
Airdoc LLC（鹰瞳科技）；Monash University（莫纳什大学）；Harbin Engineering University（哈尔滨工程大学）；The University of Sydney（悉尼大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
In the real world, medical datasets often exhibit a long-tailed data distribution (i.e., a few classes occupy most of the data, while most classes have rarely few samples), which results in a challenging imbalance learning scenario. For example, there are estimated more than 40 different kinds of retinal diseases with variable morbidity, however with more than 30+ conditions are very rare from the global patient cohorts, which results in a typical long-tailed learning problem for deep learning-based screening models. In this study, we propose class subset learning by dividing the long-tailed d…
Knowledge distillation (KD) achieves promising results on the challenging problem of unsupervised anomaly detection (AD).The representation discrepancy of anomalies in the teacher-student (T-S) model provides essential evidence for AD. However, using similar or identical architectures to build the teacher and student models in previous studies hinders the diversity of anomalous representations. To tackle this problem, we propose a novel T-S model consisting of a teacher encoder and a student decoder and introduce a simple yet effective "reverse distillation" paradigm accordingly. Instead of re…
KD-OCT: Efficient Knowledge Distillation for Clinical-Grade Retinal OCT Classification
Artificial Intelligence Department University of Isfahan（人工智能系伊斯法罕大学）；Labbafinejad Hospital Shahid Beheshti University of Medical Science（拉巴法尼贾医院谢赫·贝赫什提医学科学大学）；Department of Mechanical Engineering Isfahan University of Technology（机械工程系伊斯法罕技术大学）
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education（教育部智能仿生无人系统重点实验室）；School of Artificial Intelligence, University of Science and Technology Beijing（北京科技大学人工智能学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
Unsupervised domain adaptation has been proposed recently to tackle the so-called domain shift between training data and test data with different distributions. However, most of them only focus on single-target domain adaptation and cannot be applied to the scenario with multiple target domains. In this paper, we propose RVms, a novel unsupervised multi-target domain adaptation approach to segment retinal vessels (RVs) from multimodal and multicenter retinal images. RVms mainly consists of a style augmentation and transfer (SAT) module and a dual-teacher knowledge distillation (DTKD) module. S…
University of Nevada, Reno（内华达大学里诺分校）；Houston Eye Associates（休斯顿眼科协会）
Ophthalmic images may contain identical-looking pathologies that can cause failure in automated techniques to distinguish different retinal degenerative diseases. Additionally, reliance on large annotated datasets and lack of knowledge distillation can restrict ML-based clinical support systems' deployment in real-world environments. To improve the robustness and transferability of knowledge, an enhanced feature-learning module is required to extract meaningful spatial representations from the retinal subspace. Such a module, if used effectively, can detect unique disease traits and differenti…
In the real world, medical datasets often exhibit a long-tailed data distribution (i.e., a few classes occupy the majority of the data, while most classes have only a limited number of samples), which results in a challenging long-tailed learning scenario. Some recently published datasets in ophthalmology AI consist of more than 40 kinds of retinal diseases with complex abnormalities and variable morbidity. Nevertheless, more than 30 conditions are rarely seen in global patient cohorts. From a modeling perspective, most deep learning models trained on these datasets may lack the ability to gen…
Unsupervised representation learning has proved to be a critical component of anomaly detection/localization in images. The challenges to learn such a representation are two-fold. Firstly, the sample size is not often large enough to learn a rich generalizable representation through conventional techniques. Secondly, while only normal samples are available at training, the learned features should be discriminative of normal and anomalous samples. Here, we propose to use the "distillation" of features at various layers of an expert network, pre-trained on ImageNet, into a simpler cloner network…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Stein Eye Institute（斯坦眼科研究所）；VoxelCloud, Inc.（体素云公司）
The retinal vasculature provides important clues in the diagnosis and monitoring of systemic diseases including hypertension and diabetes. The microvascular system is of primary involvement in such conditions, and the retina is the only anatomical site where the microvasculature can be directly observed. The objective assessment of retinal vessels has long been considered a surrogate biomarker for systemic vascular diseases, and with recent advancements in retinal imaging and computer vision technologies, this topic has become the subject of renewed attention. In this paper, we present a novel…
MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images
Hong Kong University of Science and Technology（香港科技大学）；Guangdong Weiren Meditech Co., Ltd.（广东伟仁医疗科技有限公司）；Yunnan United Vision Innovations Technology Co., Ltd.（云南联合视力创新科技有限公司）；Guangdong Hospital of Integrated Traditional Chinese and Western Medicine（广东省中西医结合医院）；Second People’s Hospital of Foshan（佛山市第二人民医院）
University of New South Wales（新南威尔士大学）；Jiangnan University（江南大学）；University of Technology Sydney（悉尼科技大学）
Open-World Compositional Zero-Shot Learning (OW-CZSL) aims to recognize new compositions of seen attributes and objects. In OW-CZSL, methods built on the conventional closed-world setting degrade severely due to the unconstrained OW test space. While previous works alleviate the issue by pruning compositions according to external knowledge or correlations in seen pairs, they introduce biases that harm the generalization. Some methods thus predict state and object with independently constructed and trained classifiers, ignoring that attributes are highly context-dependent and visually entangled…
Hangzhou Normal University（杭州师范大学）；Zhejiang University（浙江大学）
Existing state-of-the-art human pose estimation methods require heavy computational resources for accurate predictions. One promising technique to obtain an accurate yet lightweight pose estimator is knowledge distillation, which distills the pose knowledge from a powerful teacher model to a less-parameterized student model. However, existing pose distillation works rely on a heavy pre-trained estimator to perform knowledge transfer and require a complex two-stage learning procedure. In this work, we investigate a novel Online Knowledge Distillation framework by distilling Human Pose structure…
Fundus-Enhanced Disease-Aware Distillation Model for Retinal Disease Classification from OCT Images
The Hong Kong University of Science and Technology（香港科技大学）；Guangdong Provincial Hospital of Integrated Traditional Chinese and Western Medicine（广东省中西医结合医院）；Guangdong Weiren Meditech Co., Ltd（广东唯仁医疗科技有限公司）
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Institute of Artificial Intelligence, Shaoxing University（绍兴市文理学院人工智能学院）；National University of Defense Technology（国防科技大学）；Cyberspace Institute of Advanced Technology, Guangzhou University（广州大学网络空间先进技术研究院）
Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Peking University（北京大学）；Alibaba Group（阿里巴巴集团）
Lightweight Self-Knowledge Distillation with Multi-source Information Fusion
Northeastern University at Qinhuangdao（东北大学秦皇岛分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Toward Realistic Remote Sensing Dataset Distillation with Discriminative Prototype-guided Diffusion
Computer Vision and Learning Systems, Department of Electrical Engineering, Linköping University（计算机视觉与学习系统，电气工程系，利厄普大学）；Helmholtz-Zentrum Dresden-Rossendorf, Responsible AI Group（海姆霍尔茨·德累斯顿-罗斯托克研究中心，负责任人工智能小组）；University of Iceland, Faculty of Electrical and Computer Engineering（冰岛大学，电气与计算机工程学院）
Clinical Graph-Mediated Distillation for Unpaired MRI-to-CFI Hypertension Prediction
Department of Computer Science and Engineering（计算机科学与工程系）；Research Convergence Institute（研究融合院）；Department of AI Systems Engineering（人工智能系统工程系）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Optical coherence tomography (OCT) is a prevalent non-invasive imaging method which provides high resolution volumetric visualization of retina. However, its inherent defect, the speckle noise, can seriously deteriorate the tissue visibility in OCT. Deep learning based approaches have been widely used for image restoration, but most of these require a noise-free reference image for supervision. In this study, we present a diffusion probabilistic model that is fully unsupervised to learn from noise instead of signal. A diffusion process is defined by adding a sequence of Gaussian noise to self-…
Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities
School of Computer Science and Engineering, Southeast University, China（计算机科学与工程学院，东南大学，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China（新一代人工智能技术及其交叉应用重点实验室，教育部，中国）；Nanjing University of Science and Technology, China（南京理工大学，中国）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shandong University of Science and Technology（山东科技大学）
Efficient medical image segmentation aims to provide accurate pixel-wise predictions for medical images with a lightweight implementation framework. However, lightweight frameworks generally fail to achieve superior performance and suffer from poor generalizable ability on cross-domain tasks. In this paper, we explore the generalizable knowledge distillation for the efficient segmentation of cross-domain medical images. Considering the domain gaps between different medical datasets, we propose the Model-Specific Alignment Networks (MSAN) to obtain the domain-invariant representations. Meanwhil…
Self-supervised Pre-training Helps Retinal Disease Progression Modelling Most When Data Is Scarce
Hertie Institute for AI in Brain Health, Faculty of Medicine, University of Tübingen（赫蒂人工智能脑健康研究所，图宾根大学医学院）；Tübingen AI Center, University of Tübingen（图宾根人工智能中心，图宾根大学）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Bio and Brain Engineering（生物与脑工程系）；Kim Jaechul Graduate School of AI（金在哲人工智能研究生院）；Brain and Cognitive Sciences（脑与认知科学系）
Distribution Matching Distillation without Fake Score Network
Department of Computer Science and Engineering, Seoul National University（首尔国立大学计算机科学与工程系）；Department of Computer Science and Engineering, Sogang University（成均馆大学计算机科学与工程系）
To boost the performance, deep neural networks require deeper or wider network structures that involve massive computational and memory costs. To alleviate this issue, the self-knowledge distillation method regularizes the model by distilling the internal knowledge of the model itself. Conventional self-knowledge distillation methods require additional trainable parameters or are dependent on the data. In this paper, we propose a simple and effective self-knowledge distillation method using a dropout (SD-Dropout). SD-Dropout distills the posterior distributions of multiple models through a dro…
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；Capital Medical University（首都医科大学）；Deepwise AI Lab（深睿医疗AI实验室）；Beijing Stars Universal Technology Co Ltd（北京星辰万联科技有限公司）
Forecasting Parapapillary atrophy (PPA), i.e., a symptom related to most irreversible eye diseases, provides an alarm for implementing an intervention to slow down the disease progression at early stage. A key question for this forecast is: how to fully utilize the historical data (e.g., retinal image) up to the current stage for future disease prediction? In this paper, we provide an answer with a novel framework, namely \textbf{D}isease \textbf{F}orecast via \textbf{P}rogression \textbf{L}earning (\textbf{DFPL}), which exploits the irreversibility prior (i.e., cannot be reversed once diagnos…
Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis
Sun Yat-Sen University（中山大学）；ByteDance Seed Vision（字节跳动种子视觉）；Guangdong Provincial Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China（教育部机器智能与高级计算重点实验室）；Pazhou Lab (HuangPu), Guangzhou, China（琶洲实验室（黄埔），广州，中国）
DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery
University of Electronic Science and Technology of China, Chengdu, China（电子科技大学，成都，中国）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore（高性能计算研究所（IHPC），科技研究局（A*STAR），新加坡）；Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore（前沿人工智能研究中心（CFAR），科技研究局（A*STAR），新加坡）
