Physics-Driven Autoregressive State Space Models for Medical Image Reconstruction
Dept. of Electrical and Electronics Engineering, and National Magnetic Resonance Research Center (UMRAM), Bilkent University（电气与电子工程系，国家磁共振研究中心（UMRAM），比尔肯大学）
Bilal Kabas, Fuat Arslan, Valiyeh A. Nezhad, Saban Ozturk, Emine U. Saritas, Tolga Çukur
Medical image reconstruction from undersampled acquisitions is an ill-posed inverse problem requiring accurate recovery of anatomical structures from incomplete measurements. Physics-driven (PD) network models have gained prominence for this task by integrating data-consistency mechanisms with learned priors, enabling improved performance over purely data-driven approaches. However, reconstruction quality still hinges on the network's ability to disentangle artifacts from true anatomical signals-both of which exhibit complex, multi-scale contextual structure. Convolutional neural networks (CNNs) capture local correlations but often struggle with non-local dependencies. While transformers aim to alleviate this limitation, practical implementations involve design compromises to reduce computational cost by balancing local and non-local sensitivity, occasionally resulting in performance comparable to CNNs. To address these challenges, we propose MambaRoll, a novel physics-driven autoregressive state space model (SSM) for high-fidelity and efficient image reconstruction. MambaRoll employs an unrolled architecture where each cascade autoregressively predicts finer-scale feature maps conditioned on coarser-scale representations, enabling consistent multi-scale context propagation. Each stage is built on a hierarchy of scale-specific PD-SSM modules that capture spatial dependencies while enforcing data consistency through residual correction. To further improve scale-aware learning, we introduce a Deep Multi-Scale Decoding (DMSD) loss, which provides supervision at intermediate spatial scales in alignment with the autoregressive design. Demonstrations on accelerated MRI and sparse-view CT reconstructions show that MambaRoll consistently outperforms state-of-the-art CNN-, transformer-, and SSM-based methods.
Peking University（北京大学）；Academy for Advanced Interdisciplinary Studies（前沿交叉学科研究院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
We propose a novel and unified method, measurement-conditioned denoising diffusion probabilistic model (MC-DDPM), for under-sampled medical image reconstruction based on DDPM. Different from previous works, MC-DDPM is defined in measurement domain (e.g. k-space in MRI reconstruction) and conditioned on under-sampling mask. We apply this method to accelerate MRI reconstruction and the experimental results show excellent performance, outperforming full supervision baseline and the state-of-the-art score-based reconstruction method. Due to its generative nature, MC-DDPM can also quantify the unce…
The Athenian School（雅典纳中学）；Cornell University（康奈尔大学）
Because of the necessity to obtain high-quality images with minimal radiation doses, such as in low-field magnetic resonance imaging, super-resolution reconstruction in medical imaging has become more popular (MRI). However, due to the complexity and high aesthetic requirements of medical imaging, image super-resolution reconstruction remains a difficult challenge. In this paper, we offer a deep learning-based strategy for reconstructing medical images from low resolutions utilizing Transformer and Generative Adversarial Networks (T-GAN). The integrated system can extract more precise texture …
We propose and evaluate a new MRI reconstruction method named LORAKI that trains an autocalibrated scan-specific recurrent neural network (RNN) to recover missing k-space data. Methods like GRAPPA, SPIRiT, and AC-LORAKS assume that k-space data has shift-invariant autoregressive structure, and that the scan-specific autoregression relationships needed to recover missing samples can be learned from fully-sampled autocalibration (ACS) data. Recently, the structure of the linear GRAPPA method has been translated into a nonlinear deep learning method named RAKI. RAKI uses ACS data to train an arti…
MambaMIR: An Arbitrary-Masked Mamba for Joint Medical Image Reconstruction and Uncertainty Estimation
Imperial College London（伦敦帝国学院）；Royal Brompton Hospital（皇家布朗普顿医院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Cambridge（剑桥大学）
Enhancing Global Sensitivity and Uncertainty Quantification in Medical Image Reconstruction with Monte Carlo Arbitrary-Masked Mamba
Imperial College London（帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）；College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区生物医学工程学院）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；Inselspital, University of Bern（伯尔尼大学因塞尔医院）；Technical University of Munich（慕尼黑工业大学）
Distillation Learning Guided by Image Reconstruction for One-Shot Medical Image Segmentation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Peking University Third Hospital（北京大学第三医院）；Beijing Children’s Hospital, Capital Medical University（首都医科大学附属北京儿童医院）；Duke University（杜克大学）
Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder
Harvard University（哈佛大学）；Singapore Management University（新加坡管理大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；South Australian Health and Medical Research Institute（南澳大利亚健康与医学研究所）；University of Surrey（萨里大学）
University of Cambridge（剑桥大学）；Rice University（莱斯大学）
The Magnetic Resonance Imaging (MRI) processing chain starts with a critical acquisition stage that provides raw data for reconstruction of images for medical diagnosis. This flow usually includes a near-lossless data compression stage that enables digital storage and/or transmission in binary formats. In this work we propose a framework for joint optimization of the MRI reconstruction and lossy compression, producing compressed representations of medical images that achieve improved trade-offs between quality and bit-rate. Moreover, we demonstrate that lossy compression can even improve the r…
National University of Singapore（新加坡国立大学）；University of Cambridge（剑桥大学）
Indirect image registration is a promising technique to improve image reconstruction quality by providing a shape prior for the reconstruction task. In this paper, we propose a novel hybrid method that seeks to reconstruct high quality images from few measurements whilst requiring low computational cost. With this purpose, our framework intertwines indirect registration and reconstruction tasks is a single functional. It is based on two major novelties. Firstly, we introduce a model based on deep nets to solve the indirect registration problem, in which the inversion and registration mappings …
Imperial College London（帝国理工学院）；University of Edinburgh（爱丁堡大学）；Technical University of Munich（慕尼黑工业大学）
Artificial intelligence (AI) and Machine Learning (ML) have shown great potential in improving the medical imaging workflow, from image acquisition and reconstruction to disease diagnosis and treatment. Particularly, in recent years, there has been a significant growth in the use of AI and ML algorithms, especially Deep Learning (DL) based methods, for medical image reconstruction. DL techniques have shown to be competitive and often superior over conventional reconstruction methods in terms of both reconstruction quality and computational efficiency. The use of DL-based image reconstruction a…
The optical resolution of a digital camera is one of its most crucial parameters with broad relevance for consumer electronics, surveillance systems, remote sensing, or medical imaging. However, resolution is physically limited by the optics and sensor characteristics. In addition, practical and economic reasons often stipulate the use of out-dated or low-cost hardware. Super-resolution is a class of retrospective techniques that aims at high-resolution imagery by means of software. Multi-frame algorithms approach this task by fusing multiple low-resolution frames to reconstruct high-resolutio…
ToNNO: Tomographic Reconstruction of a Neural Network's Output for Weakly Supervised Segmentation of 3D Medical Images
TheraPanacea；University of Crete（克里特大学）；IACM, FORTH（FORTH 应用与计算数学研究所）；Archimedes, Athena RC（雅典研究中心阿基米德研究所）；CentraleSupélec, University of Paris-Saclay（中央理工-高等电力学院，巴黎-萨克雷大学）
Medical imaging is crucial in modern clinics to guide the diagnosis and treatment of diseases. Medical image reconstruction is one of the most fundamental and important components of medical imaging, whose major objective is to acquire high-quality medical images for clinical usage at minimal cost and risk to the patients. Mathematical models in medical image reconstruction or, more generally, image restoration in computer vision, have been playing a prominent role. Earlier mathematical models are mostly designed by human knowledge or hypothesis on the image to be reconstructed, and we shall c…
What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning, Technical University of Munich（慕尼黑工业大学慕尼黑机器学习中心）；Klinikum rechts der Isar, Technical University of Munich（慕尼黑工业大学伊萨尔河右岸医院）；ELLIS Institute Finland（芬兰ELLIS研究所）；Aalto University（阿尔托大学）；Imperial College London（伦敦帝国理工学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Computed medical imaging systems require a computational reconstruction procedure for image formation. In order to recover a useful estimate of the object to-be-imaged when the recorded measurements are incomplete, prior knowledge about the nature of object must be utilized. In order to improve the conditioning of an ill-posed imaging inverse problem, deep learning approaches are being actively investigated for better representing object priors and constraints. This work proposes to use a style-based generative adversarial network (StyleGAN) to constrain an image reconstruction problem in the …
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局资讯通信研究院）；Southern University of Science and Technology（南方科技大学）；Cixi Institute of Biomedical Engineering, Chinese Academy of Sciences（中国科学院慈溪生物医学工程研究所）
Anomaly detection in medical images refers to the identification of abnormal images with only normal images in the training set. Most existing methods solve this problem with a self-reconstruction framework, which tends to learn an identity mapping and reduces the sensitivity to anomalies. To mitigate this problem, in this paper, we propose a novel Proxy-bridged Image Reconstruction Network (ProxyAno) for anomaly detection in medical images. Specifically, we use an intermediate proxy to bridge the input image and the reconstructed image. We study different proxy types, and we find that the sup…
The idea of federated learning is to train deep neural network models collaboratively and share them with multiple participants without exposing their private training data to each other. This is highly attractive in the medical domain due to patients' privacy records. However, a recently proposed method called Deep Leakage from Gradients enables attackers to reconstruct data from shared gradients. This study shows how easy it is to reconstruct images for different data initialization schemes and distance measures. We show how data and model architecture influence the optimal choice of initial…
Tackling Hallucination from Conditional Models for Medical Image Reconstruction with DynamicDPS
Hawkes Institute, University College London, UK Department of Medical Physics；Biomedical Engineering, University College London, UK Department of Computer Science, University College London, UK The University of Utah, USA, Centre for AI, DS\&AI, AstraZeneca, UK
NVIDIA（英伟达）；DeepMind（深度思维）；Oxford Nanopore Technologies（牛津纳米孔技术公司）；University College London（伦敦大学学院）
Human anatomy, morphology, and associated diseases can be studied using medical imaging data. However, access to medical imaging data is restricted by governance and privacy concerns, data ownership, and the cost of acquisition, thus limiting our ability to understand the human body. A possible solution to this issue is the creation of a model able to learn and then generate synthetic images of the human body conditioned on specific characteristics of relevance (e.g., age, sex, and disease status). Deep generative models, in the form of neural networks, have been recently used to create synthe…
In this paper we broadly consider techniques which utilize projections on rays for data collection, with particular emphasis on optical techniques. We formulate a variety of imaging techniques as either special cases or extensions of tomographic reconstruction. We then consider how the techniques must be extended to describe objects containing occlusion, as with a self-occluding opaque object. We formulate the reconstruction problem as a regularized nonlinear optimization problem to simultaneously solve for object brightness and attenuation, where the attenuation can become infinite. We demons…
University of Rome Tor Vergata（罗马第二大学）；University of Cambridge（剑桥大学）；Athinoula A. Martinos Center for Biomedical Imaging（阿西诺拉·A·马蒂诺斯生物医学成像中心）
The lack of large labeled medical imaging datasets, along with significant inter-individual variability compared to clinically established disease classes, poses significant challenges in exploiting medical imaging information in a precision medicine paradigm, where in principle dense patient-specific data can be employed to formulate individual predictions and/or stratify patients into finer-grained groups which may follow more homogeneous trajectories and therefore empower clinical trials. In order to efficiently explore the effective degrees of freedom underlying variability in medical imag…
Are Video Models Emerging as Zero-Shot Learners and Reasoners in Medical Imaging?
Department of Computer Science, Emory University（埃默里大学计算机科学系）；Department of Computer Science, University of Southern California（南加州大学计算机科学系）；Department of Computer Science, University of Maryland（马里兰大学计算机科学系）；Department of Biomedical Engineering, Georgia Institute of Technology（佐治亚理工学院生物医学工程系）；Department of Radiation Oncology and Winship Cancer Institute, Emory University（埃默里大学放射肿瘤学系和Winship癌症研究所）
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院保罗·C·劳特伯生物医学成像研究中心）；Research center for Medical AI, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院医学人工智能研究中心）；Nanchang University（南昌大学）；University at Buffalo, the State University of New York（纽约州立大学布法罗分校）
Medical imaging is playing a more and more important role in clinics. However, there are several issues in different imaging modalities such as slow imaging speed in MRI, radiation injury in CT and PET. Therefore, accelerating MRI, reducing radiation dose in CT and PET have been ongoing research topics since their invention. Usually, acquiring less data is a direct but important strategy to address these issues. However, less acquisition usually results in aliasing artifacts in reconstructions. Recently, deep learning (DL) has been introduced in medical image reconstruction and shown potential…
Evaluating the Impact of Medical Image Reconstruction on Downstream AI Fairness and Performance
Department of Data Science, Dana-Farber Cancer Institute（丹娜-法伯癌症研究所数据科学系）；AI in Medicine, Technical University of Munich（慕尼黑工业大学医学人工智能）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Computing, Imperial College London（伦敦帝国理工学院计算系）；Harvard Medical School（哈佛医学院）
Diffusion Low Rank Hybrid Reconstruction for Sparse View Medical Imaging
Shenzhen MSU-BIT University（深圳MSU-BIT大学）；China Media Group（中国媒体集团）；School of Mathematics and Statistics, Beijing Institute of Technology（北京理工大学数学与统计学院）
Charité - Universitätsmedizin Berlin（柏林夏里特医学院）；University of Innsbruck（因斯布鲁克大学）；Physikalisch-Technische Bundesanstalt (PTB)（德国联邦物理技术研究院）；Technical University of Berlin（柏林工业大学）；German Cancer Research Center（德国癌症研究中心）
In this paper we present a generalized Deep Learning-based approach for solving ill-posed large-scale inverse problems occuring in medical image reconstruction. Recently, Deep Learning methods using iterative neural networks and cascaded neural networks have been reported to achieve state-of-the-art results with respect to various quantitative quality measures as PSNR, NRMSE and SSIM across different imaging modalities. However, the fact that these approaches employ the forward and adjoint operators repeatedly in the network architecture requires the network to process the whole images or volu…
Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Medical Image Reconstruction
University College London（伦敦大学学院）；Atinary Technologies（Atinary科技公司）；University of Bremen（不来梅大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Ajou University School of Medicine（亚洲大学医学院）；The Chinese University of Hong Kong（香港中文大学）
Multibranch Generative Models for Multichannel Imaging with an Application to PET/CT Synergistic Reconstruction
Univ. Brest, LaTIM, INSERM, UMR 1101（布雷斯特大学、LaTIM研究所、法国国家健康与医学研究院、UMR 1101联合研究单位）；Nuclear Medicine Department, Poitiers University Hospital（普瓦捷大学医院核医学科）
End-to-End Learning of Multi-Organ Implicit Surfaces from 3D Medical Imaging Data
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France Institute of Image-Guided Surgery, IHU Strasbourg, Strasbourg, France Department of Diagnostic Radiology, McGill University, Montreal, Canada Augmented Intelligence \& Precision Health Laboratory (AIPHL), McGill University Health Centre Research Institute, Montreal, Canada
Harvard University（哈佛大学）；Massachusetts General Hospital（麻省总医院）
Recent advancements in deep learning for automated image processing and classification have accelerated many new applications for medical image analysis. However, most deep learning applications have been developed using reconstructed, human-interpretable medical images. While image reconstruction from raw sensor data is required for the creation of medical images, the reconstruction process only uses a partial representation of all the data acquired. Here we report the development of a system to directly process raw computed tomography (CT) data in sinogram-space, bypassing the intermediary s…
From Redaction to Restoration: Deep Learning for Medical Image Anonymization and Reconstruction
Center for Artificial Intelligence, BCVI, Northwestern Medicine（西北医学BCVI人工智能中心）；Department of Electrical and Computer Engineering, Northwestern University（西北大学电气与计算机工程系）；Department of Surgery, Northwestern University（西北大学外科学系）；Xtasis Inc.（Xtasis公司）；Medtronic（美敦力公司）；Departments of Radiology and Clinical Neurosciences, University of Calgary（卡尔加里大学放射学与临床神经科学系）
Technical University of Berlin（柏林工业大学）；Fraunhofer HHI（弗劳恩霍夫海因里希·赫兹研究所）
This work investigates the detection of instabilities that may occur when utilizing deep learning models for image reconstruction tasks. Although neural networks often empirically outperform traditional reconstruction methods, their usage for sensitive medical applications remains controversial. Indeed, in a recent series of works, it has been demonstrated that deep learning approaches are susceptible to various types of instabilities, caused for instance by adversarial noise or out-of-distribution features. It is argued that this phenomenon can be observed regardless of the underlying archite…
FaRMamba: Frequency-based learning and Reconstruction aided Mamba for Medical Segmentation
School of Artificial Intelligence and Computer Science, Nantong University, China（人工智能与计算机科学学院，南通大学）；School of Information Science and Technology, Nantong University, China（信息科学与技术学院，南通大学）
Deep learning-based computer-aided diagnosis is gradually deployed to review and analyze medical images. However, this paradigm is restricted in real-world clinical applications due to the poor robustness and generalization. The issue is more sinister with a lack of training data. In this paper, we address the challenge from the transfer learning point of view. Different from the common setting that transferring knowledge from the natural image domain to the medical image domain, we find the knowledge from the same domain further boosts the model robustness and generalization. Therefore, we pr…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；Polish-Japanese Academy of Information Technology（波兰-日本信息技术学院）；The Maria Skłodowska-Curie Memorial Cancer Centre and Institute of Oncology（玛丽·斯克洛多夫斯卡-居里纪念癌症中心和肿瘤研究所）
Deep learning algorithms, especially convolutional neural networks, have become a methodology of choice in medical image analysis. However, recent studies in computer vision show that even a small modification of input image intensities may cause a deep learning model to classify the image differently. In medical imaging, the distribution of image intensities is related to applied image reconstruction algorithm. In this paper we investigate the impact of ultrasound image reconstruction method on breast lesion classification with neural transfer learning. Due to high dynamic range raw ultrasoni…
The combination of tomographic imaging and deep learning, or machine learning in general, promises to empower not only image analysis but also image reconstruction. The latter aspect is considered in this perspective article with an emphasis on medical imaging to develop a new generation of image reconstruction theories and techniques. This direction might lead to intelligent utilization of domain knowledge from big data, innovative approaches for image reconstruction, and superior performance in clinical and preclinical applications. To realize the full impact of machine learning on medical i…
