Non-visual imaging sensors are widely used in the industry for different purposes. Those sensors are more expensive than visual (RGB) sensors, and usually produce images with lower resolution. To this end, Cross-Modality Super-Resolution methods were introduced, where an RGB image of a high-resolution assists in increasing the resolution of the low-resolution modality. However, fusing images from different modalities is not a trivial task; the output must be artifact-free and remain loyal to the characteristics of the target modality. Moreover, the input images are never perfectly aligned, which results in further artifacts during the fusion process.
We present CMSR, a deep network for Cross-Modality Super-Resolution, which unlike previous methods, is designed to deal with weakly aligned images. The network is trained on the two input images only, learns their internal statistics and correlations, and applies them to up-sample the target modality. CMSR contains an internal transformer that is trained on-the-fly together with the up-sampling process itself, without explicit supervision. We show that CMSR succeeds to increase the resolution of the input image, gaining valuable information from its RGB counterpart, yet in a conservative way, without introducing artifacts or irrelevant details.
Deep learning methods have been successfully applied to various computer vision tasks. However, existing neural network architectures do not per se incorporate domain knowledge about the addressed problem, thus, understanding what the model has learned is an open research topic. In this paper, we rely on the unfolding of an iterative algorithm for sparse approximation with side information, and design a deep learning architecture for multimodal image super-resolution that incorporates sparse priors and effectively utilizes information from another image modality. We develop two deep models per…
Nagoya University（名古屋大学）；Sapporo-Kosei General Hospital（札幌厚生综合医院）；Sapporo Minami-sanjo Hospital（札幌南三条医院）；Keiwakai Nishioka Hospital（惠和会西冈医院）；National Institute of Informatics（国立信息学研究所）
This paper newly introduces multi-modality loss function for GAN-based super-resolution that can maintain image structure and intensity on unpaired training dataset of clinical CT and micro CT volumes. Precise non-invasive diagnosis of lung cancer mainly utilizes 3D multidetector computed-tomography (CT) data. On the other hand, we can take micro CT images of resected lung specimen in 50 micro meter or higher resolution. However, micro CT scanning cannot be applied to living human imaging. For obtaining highly detailed information such as cancer invasion area from pre-operative clinical CT vol…
Improving the resolution of magnetic resonance (MR) image data is critical to computer-aided diagnosis and brain function analysis. Higher resolution helps to capture more detailed content, but typically induces to lower signal-to-noise ratio and longer scanning time. To this end, MR image super-resolution has become a widely-interested topic in recent times. Existing works establish extensive deep models with the conventional architectures based on convolutional neural networks (CNN). In this work, to further advance this research field, we make an early effort to build a Transformer-based MR…
Nagoya University（名古屋大学）；Nagoya University Graduate School of Medicine（名古屋大学医学研究科）
This paper proposes a novel, unsupervised super-resolution (SR) approach for performing the SR of a clinical CT into the resolution level of a micro CT ($μ$CT). The precise non-invasive diagnosis of lung cancer typically utilizes clinical CT data. Due to the resolution limitations of clinical CT (about $0.5 \times 0.5 \times 0.5$ mm$^3$), it is difficult to obtain enough pathological information such as the invasion area at alveoli level. On the other hand, $μ$CT scanning allows the acquisition of volumes of lung specimens with much higher resolution ($50 \times 50 \times 50 μ{\rm m}^3$ or hig…
The University of Sheffield（谢菲尔德大学）；University of East Anglia（东英吉利大学）
Cross-modal image synthesis is a topical problem in medical image computing. Existing methods for image synthesis are either tailored to a specific application, require large scale training sets, or are based on partitioning images into overlapping patches. In this paper, we propose a novel Dual cOnvolutional filTer lEarning (DOTE) approach to overcome the drawbacks of these approaches. We construct a closed loop joint filter learning strategy that generates informative feedback for model self-optimization. Our method can leverage data more efficiently thus reducing the size of the required tr…
Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality. While existing deep multimodal models do not incorporate domain knowledge about image SR, we present a multimodal deep network design that integrates coupled sparse priors and allows the effective fusion of information from another modality into the reconstruction process. Our method is inspired by a novel iterative algorithm for coupled convolutional sparse coding, resulting in an interpretable network by design. We apply our model …
DeepLight: Reconstructing High-Resolution Observations of Nighttime Light With Multi-Modal Remote Sensing Data
Tsinghua University - Xi’an Institute of Surveying and Mapping Joint Research Center（清华大学-西安测绘联合研究中心）；National supercomputing center in Shenzhen, Tsinghua University（清华大学深圳国家超级计算中心）；The University of Hong Kong（香港大学）；Sun-Yat San University（中山大学）
Northwestern Polytechnical University（西北工业大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai United Imaging Intelligence Co., Ltd（上海联影智能有限公司）；Shanghai Jiao Tong University（上海交通大学）
Non-rigid inter-modality registration can facilitate accurate information fusion from different modalities, but it is challenging due to the very different image appearances across modalities. In this paper, we propose to train a non-rigid inter-modality image registration network, which can directly predict the transformation field from the input multimodal images, such as CT and MR images. In particular, the training of our inter-modality registration network is supervised by intra-modality similarity metric based on the available paired data, which is derived from a pre-aligned CT and MR da…
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education（教育部高维信息智能感知与系统重点实验室 PCA 实验室）；Jiangsu Key Lab of Image and Video Understanding for Social Security（江苏省社会安全图像与视频理解重点实验室）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Real depth super-resolution (DSR), unlike synthetic settings, is a challenging task due to the structural distortion and the edge noise caused by the natural degradation in real-world low-resolution (LR) depth maps. These defeats result in significant structure inconsistency between the depth map and the RGB guidance, which potentially confuses the RGB-structure guidance and thereby degrades the DSR quality. In this paper, we propose a novel structure flow-guided DSR framework, where a cross-modality flow map is learned to guide the RGB-structure information transferring for precise depth upsa…
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Northwestern Polytechnical University（西北工业大学）；Terminus Group（特斯联集团）；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）
Accelerated multi-modal magnetic resonance (MR) imaging is a new and effective solution for fast MR imaging, providing superior performance in restoring the target modality from its undersampled counterpart with guidance from an auxiliary modality. However, existing works simply combine the auxiliary modality as prior information, lacking in-depth investigations on the potential mechanisms for fusing different modalities. Further, they usually rely on the convolutional neural networks (CNNs), which is limited by the intrinsic locality in capturing the long-distance dependency. To this end, we …
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所先进智能项目中心）；National University of Defense Technology（国防科技大学）；Hitachi, Ltd（日立制作所）
Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In t…
Over the past few years deep learning-based techniques such as Generative Adversarial Networks (GANs) have significantly improved solutions to image super-resolution and image-to-image translation problems. In this paper, we propose a solution to the joint problem of image super-resolution and multi-modality image-to-image translation. The problem can be stated as the recovery of a high-resolution image in a modality, given a low-resolution observation of the same image in an alternative modality. Our paper offers two models to address this problem and will be evaluated on the recovery of high…
Modality-Aware Infrared and Visible Image Fusion with Target-Aware Supervision
Independent researcher（独立研究者）；Dept. of Computer Science Baylor University（计算机科学系 巴里尔大学）；Dept. of Computer Science Engineering University of Connecticut（计算机科学工程系 佛罗里达大学）；Dept. of Computer Science New York University（计算机科学系 新 york 大学）
Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province（辽宁省泛在网络与服务软件重点实验室）；Baidu Research（百度研究院）
Existing color-guided depth super-resolution (DSR) approaches require paired RGB-D data as training samples where the RGB image is used as structural guidance to recover the degraded depth map due to their geometrical similarity. However, the paired data may be limited or expensive to be collected in actual testing environment. Therefore, we explore for the first time to learn the cross-modality knowledge at training stage, where both RGB and depth modalities are available, but test on the target dataset, where only single depth modality exists. Our key idea is to distill the knowledge of scen…
multimodars: A Rust-powered toolkit for multi-modality cardiac image fusion and registration
Department of Cardiology, Inselspital, Bern University Hospital, University of Bern（伯尔尼大学伯尔尼大学医院 Inselspital 心脏病学系）；Graduate School for Cellular and Biomedical Sciences, University of Bern（伯尔尼大学细胞与生物医学科学研究生院）
We study joint video and language (VL) pre-training to enable cross-modality learning and benefit plentiful downstream VL tasks. Existing works either extract low-quality video features or learn limited text embedding, while neglecting that high-resolution videos and diversified semantics can significantly improve cross-modality learning. In this paper, we propose a novel High-resolution and Diversified VIdeo-LAnguage pre-training model (HD-VILA) for many visual tasks. In particular, we collect a large dataset with two distinct properties: 1) the first high-resolution dataset including 371.5k …
Multi-modality image registration is one of the most underlined processes in medical image analysis. Recently, convolutional neural networks (CNNs) have shown significant potential in deformable registration. However, the lack of voxel-wise ground truth challenges the training of CNNs for an accurate registration. In this work, we propose a cross-modality similarity metric, based on the KL-divergence of image variables, and implement an efficient estimation method using a CNN. This estimation network, referred to as KLDivNet, can be trained unsupervisedly. We then embed the KLDivNet into a reg…
Istanbul Technical University（伊斯坦布尔理工大学）；Université de Sousse（苏塞大学）；Ecole Nationale d’Ingénieurs de Sousse（苏塞国立工程师学院）
Synthesizing multimodality medical data provides complementary knowledge and helps doctors make precise clinical decisions. Although promising, existing multimodal brain graph synthesis frameworks have several limitations. First, they mainly tackle only one problem (intra- or inter-modality), limiting their generalizability to synthesizing inter- and intra-modality simultaneously. Second, while few techniques work on super-resolving low-resolution brain graphs within a single modality (i.e., intra), inter-modality graph super-resolution remains unexplored though this would avoid the need for c…
The High-Resolution Transformer (HRFormer) can maintain high-resolution representation and share global receptive fields. It is friendly towards salient object detection (SOD) in which the input and output have the same resolution. However, two critical problems need to be solved for two-modality SOD. One problem is two-modality fusion. The other problem is the HRFormer output's fusion. To address the first problem, a supplementary modality is injected into the primary modality by using global optimization and an attention mechanism to select and purify the modality at the input level. To solv…
MoCoLSK: Modality Conditioned High-Resolution Downscaling for Land Surface Temperature
Nanjing University of Science and Technology（南京理工大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Nankai University（南开大学）；Guangzhou Institute of Geography, Guangdong Academy of Sciences（广东省科学院广州地理研究所）
The University of Sheffield（谢菲尔德大学）；University of East Anglia（东英吉利大学）
Magnetic Resonance Imaging (MRI) offers high-resolution \emph{in vivo} imaging and rich functional and anatomical multimodality tissue contrast. In practice, however, there are challenges associated with considerations of scanning costs, patient comfort, and scanning time that constrain how much data can be acquired in clinical or research studies. In this paper, we explore the possibility of generating high-resolution and multimodal images from low-resolution single-modality imagery. We propose the weakly-supervised joint convolutional sparse coding to simultaneously solve the problems of sup…
Global and Local Mamba Network for Multi-Modality Medical Image Super-Resolution
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University（中南大学机器视觉与智能医学湖南省工程研究中心）；University of Rouen-Normandy（鲁昂-诺曼底大学）；Henri Becquerel Cancer Center（亨利·贝克勒尔癌症中心）
Harvard University（哈佛大学）；Harvard Medical School（哈佛医学院）；Boston Children’s Hospital（波士顿儿童医院）；ESRF, The European Synchrotron（欧洲同步辐射装置）
Comprehensive, synapse-resolution imaging of the brain will be crucial for understanding neuronal computations and function. In connectomics, this has been the sole purview of volume electron microscopy (EM), which entails an excruciatingly difficult process because it requires cutting tissue into many thin, fragile slices that then need to be imaged, aligned, and reconstructed. Unlike EM, hard X-ray imaging is compatible with thick tissues, eliminating the need for thin sectioning, and delivering fast acquisition, intrinsic alignment, and isotropic resolution. Unfortunately, current state-of-…
UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation
LIGM, Ecole Nationale des Ponts et Chaussées, IP Paris, Univ Gustave Eiffel, CNRS（LIGM，巴黎国立桥梁与道路学院，IP巴黎，埃夫尔大学，CNRS）；LASTIG, Univ Gustave Eiffel, IGN, ENSG（LASTIG，埃夫尔大学，国家地理信息学院，ENSG）；IGN（国家地理信息学院）；CNES（国家航天中心）；EFEO（埃克塞特东方研究院）
Dual Modality Prompted Diffusion Priors for Zero Shot Hyperspectral Pansharpening
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Peking University（北京大学）；National Center for Applied Mathematics Shenzhen (NCAMS), Southern University of Science and Technology（南方科技大学深圳应用数学国家中心）；Department of Automation, Tsinghua University（清华大学自动化系）；National Research Council of Italy, Institute of Integrated Methodologies for Earth Observation (CNR-IMIOT)（意大利国家研究委员会综合地球观测方法研究所）
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Glaucoma is one of the ophthalmic diseases that may cause blindness, for which early detection and treatment are very important. Fundus images and optical coherence tomography (OCT) images are both widely-used modalities in diagnosing glaucoma. However, existing glaucoma grading approaches mainly utilize a single modality, ignoring the complementary information between fundus and OCT. In this paper, we propose an efficient multi-modality supervised contrastive learning framework, named COROLLA, for glaucoma grading. Through layer segmentation as well as thickness calculation and projection, re…
The Koos grading scale is a classification system for vestibular schwannoma (VS) used to characterize the tumor and its effects on adjacent brain structures. The Koos classification captures many of the characteristics of treatment deci-sions and is often used to determine treatment plans. Although both contrast-enhanced T1 (ceT1) scanning and high-resolution T2 (hrT2) scanning can be used for Koos Classification, hrT2 scanning is gaining interest because of its higher safety and cost-effectiveness. However, in the absence of annotations for hrT2 scans, deep learning methods often inevitably s…
Institute of Command and Control Engineering, Army Engineering University of PLA（中国人民解放军陆军工程大学指挥控制工程学院）；The 63rd Institute, National University of Defense Technology（国防科技大学第六十三研究所）
Multi-modal data is becoming more common in big data background. Finding the semantically similar objects from different modality is one of the heart problems of multi-modal learning. Most of the current methods try to learn the inter-modal correlation with extrinsic supervised information, while intrinsic structural information of each modality is neglected. The performance of these methods heavily depends on the richness of training samples. However, obtaining the multi-modal training samples is still a labor and cost intensive work. In this paper, we bring a extrinsic correlation between th…
Muroran Institute of Technology（室兰工业大学）；Niigata University of Health and Welfare（新潟医疗福祉大学）
To promote the development of medical image segmentation technology, AMOS, a large-scale abdominal multi-organ dataset for versatile medical image segmentation, is provided and AMOS 2022 challenge is held by using the dataset. In this report, we present our solution for the AMOS 2022 challenge. We employ residual U-Net with deep super vision as our base model. The experimental results show that the mean scores of Dice similarity coefficient and normalized surface dice are 0.8504 and 0.8476 for CT only task and CT/MRI task, respectively.
Combining the respective advantages of cross-modality images can compensate for the lack of information in the single modality, which has attracted increasing attention of researchers into multi-modal image matching tasks. Meanwhile, due to the great appearance differences between cross-modality image pairs, it often fails to make the feature representations of correspondences as close as possible. In this letter, we design a cross-modality feature representation learning network, S2-Net, which is based on the recently successful detect-and-describe pipeline, originally proposed for visible im…
Extended Cross-Modality United Learning for Unsupervised Visible-Infrared Person Re-identification
School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息工程学院）；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（测绘遥感信息工程国家重点实验室，武汉大学）；Electronic Information School, Wuhan University（电子信息学院，武汉大学）
School of Mechanical and Electric Engineering, Guangzhou University（广州大学机械与电气工程学院）；School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）
Cross-modality person re-identification is a challenging problem which retrieves a given pedestrian image in RGB modality among all the gallery images in infrared modality. The task can address the limitation of RGB-based person Re-ID in dark environments. Existing researches mainly focus on enlarging inter-class differences of feature to solve the problem. However, few studies investigate improving intra-class cross-modality similarity, which is important for this issue. In this paper, we propose a novel loss function, called Hetero-Center loss (HC loss) to reduce the intra-class cross-modali…
RMIT University（皇家墨尔本理工大学）；University of New South Wales（新南威尔士大学）；Philips Research（飞利浦研究院）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）
Self-Supervised Learning (SSL) is a new paradigm for learning discriminative representations without labelled data and has reached comparable or even state-of-the-art results in comparison to supervised counterparts. Contrastive Learning (CL) is one of the most well-known approaches in SSL that attempts to learn general, informative representations of data. CL methods have been mostly developed for applications in computer vision and natural language processing where only a single sensor modality is used. A majority of pervasive computing applications, however, exploit data from a range of dif…
Southern University of Science and Technology（南方科技大学）；University of Surrey（萨里大学）；Jarvis Research Center, Tencent YouTu Lab（腾讯优图实验室贾维斯研究中心）；University of Chinese Academy of Sciences（中国科学院大学）；Bielefeld University（比勒费尔德大学）
Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signals
Shanghai Jiao Tong University（上海交通大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of Southern California（南加州大学）；Shanghai AI Laboratory（上海人工智能实验室）
With the advances of deep learning, many medical image segmentation studies achieve human-level performance when in fully supervised condition. However, it is extremely expensive to acquire annotation on every data in medical fields, especially on magnetic resonance images (MRI) that comprise many different contrasts. Unsupervised methods can alleviate this problem; however, the performance drop is inevitable compared to fully supervised methods. In this work, we propose a self-training based unsupervised-learning framework that performs automatic segmentation of Vestibular Schwannoma (VS) and…
Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation
College of Computer Science, Sichuan University（四川大学计算机学院）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；College of Electrical Engineering, Sichuan University（四川大学电气工程学院）；School of Cyber Science and Engineering, Sichuan University（四川大学网络空间安全学院）
