Unsupervised domain adaptation has attracted growing research attention on semantic segmentation. However, 1) most existing models cannot be directly applied into lesions transfer of medical images, due to the diverse appearances of same lesion among different datasets; 2) equal attention has been paid into all semantic representations instead of neglecting irrelevant knowledge, which leads to negative transfer of untransferable knowledge. To address these challenges, we develop a new unsupervised semantic transfer model including two complementary modules (i.e., T_D and T_F ) for endoscopic lesions segmentation, which can alternatively determine where and how to explore transferable domain-invariant knowledge between labeled source lesions dataset (e.g., gastroscope) and unlabeled target diseases dataset (e.g., enteroscopy). Specifically, T_D focuses on where to translate transferable visual information of medical lesions via residual transferability-aware bottleneck, while neglecting untransferable visual characterizations. Furthermore, T_F highlights how to augment transferable semantic features of various lesions and automatically ignore untransferable representations, which explores domain-invariant knowledge and in return improves the performance of T_D. To the end, theoretical analysis and extensive experiments on medical endoscopic dataset and several non-medical public datasets well demonstrate the superiority of our proposed model.
Registration-Grounded Spectral Fusion for Unregistered WLI/NBI Endoscopic Lesion Segmentation
The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）；School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Chinese PLA General Hospital（中国人民解放军总医院）；University of Arkansas at Little Rock（阿肯色大学小石城分校）；Indiana University-Purdue University Indianapolis（印第安纳大学-普渡大学印第安纳波利斯分校）
Weakly-supervised learning has attracted growing research attention on medical lesions segmentation due to significant saving in pixel-level annotation cost. However, 1) most existing methods require effective prior and constraints to explore the intrinsic lesions characterization, which only generates incorrect and rough prediction; 2) they neglect the underlying semantic dependencies among weakly-labeled target enteroscopy diseases and fully-annotated source gastroscope lesions, while forcefully utilizing untransferable dependencies leads to the negative performance. To tackle above issues, …
Tecnologico de Monterrey（蒙特雷理工学院）；University of Leeds（利兹大学）
Prevalence of gastrointestinal (GI) cancer is growing alarmingly every year leading to a substantial increase in the mortality rate. Endoscopic detection is providing crucial diagnostic support, however, subtle lesions in upper and lower GI are quite hard to detect and cause considerable missed detection. In this work, we leverage deep learning to develop a framework to improve the localization of difficult to detect lesions and minimize the missed detection rate. We propose an end to end student-teacher learning setup where class probabilities of a trained teacher model on one class with larg…
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Pancreas Centre, First Affiliated Hospital, Nanjing Medical University（南京医科大学第一附属医院胰腺中心）；Department of General Surgery, First Affiliated Hospital, Nanjing Medical University（南京医科大学第一附属医院普外科）；Big Data Computing Center of Southeast University（东南大学大数据计算中心）
While data-driven approaches excel at many image analysis tasks, the performance of these approaches is often limited by a shortage of annotated data available for training. Recent work in semi-supervised learning has shown that meaningful representations of images can be obtained from training with large quantities of unlabeled data, and that these representations can improve the performance of supervised tasks. Here, we demonstrate that an unsupervised jigsaw learning task, in combination with supervised training, results in up to a 9.8% improvement in correctly classifying lesions in colono…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Novartis Pharmaceuticals（诺华制药）；University of California, Irvine（加州大学欧文分校）；Wolfram Research（沃尔夫勒姆研究公司）
Skin cancer, the most commonly found human malignancy, is primarily diagnosed visually via dermoscopic analysis, biopsy, and histopathological examination. However, unlike other types of cancer, automated image classification of skin lesions is deemed more challenging due to the irregularity and variability in the lesions' appearances. In this work, we propose an adaptation of the Neural Style Transfer (NST) as a novel image pre-processing step for skin lesion classification problems. We represent each dermoscopic image as the style image and transfer the style of the lesion onto a homogeneous…
University of Chinese Academy of Sciences（中国科学院大学）
Weakly-supervised learning under image-level labels supervision has been widely applied to semantic segmentation of medical lesions regions. However, 1) most existing models rely on effective constraints to explore the internal representation of lesions, which only produces inaccurate and coarse lesions regions; 2) they ignore the strong probabilistic dependencies between target lesions dataset (e.g., enteroscopy images) and well-to-annotated source diseases dataset (e.g., gastroscope images). To better utilize these dependencies, we present a new semantic lesions representation transfer model…
GIM-ENDO: A Multimodal Endoscopic Image and Video Dataset for Gastric Intestinal Metaplasia Morphology and Pathology
Gastroenterology and Liver Disease Research Center, Research Institute for Gastroenterology and Liver Diseases, Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医科大学胃肠病与肝病研究中心，胃肠病与肝病研究所）；Iran University of Medical Sciences（伊朗医科大学）；Shiraz University of Medical Sciences（设拉子医科大学）；Gastroenterology Department, Amiralam Hospital, Tehran University of Medical Sciences（德黑兰医科大学阿米拉拉姆医院胃肠病科）；Department of Computer Engineering, Science and Research Branch, Islamic Azad University（伊斯兰阿扎德大学科学与研究分校计算机工程系）
Deep learning-based Computer-Aided Diagnosis (CAD) has attracted appealing attention in academic researches and clinical applications. Nevertheless, the Convolutional Neural Networks (CNNs) diagnosis system heavily relies on the well-labeled lesion dataset, and the sensitivity to the variation of data distribution also restricts the potential application of CNNs in CAD. Unsupervised Domain Adaptation (UDA) methods are developed to solve the expensive annotation and domain gaps problem and have achieved remarkable success in medical image analysis. Yet existing UDA approaches only adapt knowled…
Politecnico di Milano（米兰理工大学）；CNRS-Université de Strasbourg（法国国家科学研究中心-斯特拉斯堡大学）；University of Calabria（卡拉布里亚大学）；Istituto Europeo di Oncologia (IRCCS)（欧洲肿瘤研究所（IRCCS））
Ureteroscopy and cystoscopy are the gold standard methods to identify and treat tumors along the urinary tract. It has been reported that during a normal procedure a rate of 10-20 % of the lesions could be missed. In this work we study the implementation of 3 different Convolutional Neural Networks (CNNs), using a 2-steps training strategy, to classify images from the urinary tract with and without lesions. A total of 6,101 images from ureteroscopy and cystoscopy procedures were collected. The CNNs were trained and tested using transfer learning in a two-steps fashion on 3 datasets. The datase…
Small intestinal capsule endoscopy is the mainstream method for inspecting small intestinal lesions,but a single small intestinal capsule endoscopy will produce 60,000 - 120,000 images, the majority of which are similar and have no diagnostic value. It takes 2 - 3 hours for doctors to identify lesions from these images. This is time-consuming and increase the probability of misdiagnosis and missed diagnosis since doctors are likely to experience visual fatigue while focusing on a large number of similar images for an extended period of time.In order to solve these problems, we proposed a simil…
Politecnico di Milano（米兰理工大学）；Université de Strasbourg（斯特拉斯堡大学）；Hospital Papa Giovanni XXIII（教皇约翰二十三世医院）
Objective: Accurate visual classification of bladder tissue during Trans-Urethral Resection of Bladder Tumor (TURBT) procedures is essential to improve early cancer diagnosis and treatment. During TURBT interventions, White Light Imaging (WLI) and Narrow Band Imaging (NBI) techniques are used for lesion detection. Each imaging technique provides diverse visual information that allows clinicians to identify and classify cancerous lesions. Computer vision methods that use both imaging techniques could improve endoscopic diagnosis. We address the challenge of tissue classification when annotation…
University College Hospital NHS Foundation Trust（伦敦大学医院 NHS 基金会信托）；National Taiwan University（国立台湾大学）；E-Da Hospital/I-Shou University（义守大学义大医院）
Purpose. Early squamous cell neoplasia (ESCN) in the oesophagus is a highly treatable condition. Lesions confined to the mucosal layer can be curatively treated endoscopically. We build a computer-assisted detection (CADe) system that can classify still images or video frames as normal or abnormal with high diagnostic accuracy. Methods. We present a new benchmark dataset containing 68K binary labeled frames extracted from 114 patient videos whose imaged areas have been resected and correlated to histopathology. Our novel convolutional network (CNN) architecture solves the binary classification…
Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis
Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education, Xiangtan University（湘潭大学教育部智能计算与信息处理重点实验室）；School of Computer Science, Xiangtan University（湘潭大学计算机科学学院）；Key Laboratory for Artificial Intelligence and International Communication, Hunan Normal University（湖南师范大学人工智能与国际传播重点实验室）
SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba
School of Biomedical Engineering（生物医学工程学院）；Division of Gastroenterology and Hepatology, Shanghai Institute of Digestive Disease, NHC Key Laboratory of Digestive Diseases, Renji Hospital（消化内科与肝病科、上海消化疾病研究所、国家消化疾病临床医学研究中心、仁济医院）；Aier Institute of Digital Ophthalmology and Visual Science（爱尔数字眼科与视觉科学研究院）；College of Media Engineering（媒体工程学院）
ERCPMP-Gx: Endoscopic Image and Video Dataset for Morphological, Histopathological, and Genomic Characterization of Colorectal Polyposis
Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医科大学）；Familial & Hereditary Cancers Institute（家族与遗传性癌症研究所）；Shiraz University of Medical Sciences（设拉子医科大学）；DataBioX Research（DataBioX研究院）
Deep learning based neural networks have gained popularity for a variety of biomedical imaging applications. In the last few years several works have shown the use of these methods for colon cancer detection and the early results have been promising. These methods can potentially be utilized to assist doctor's and may help in identifying the number of lesions or abnormalities in a diagnosis session. From our literature survey we found out that there is a lack of publicly available labeled data. Thus, as part of this work, we have aimed at open sourcing a dataset which contains annotations of p…
Polyp-Gen: Realistic and Diverse Polyp Image Generation for Endoscopic Dataset Expansion
Department of Electronic Engineering, Chinese University of Hong Kong（香港中文大学电子工程系）；Centre for Artificial Intelligence and Robotics (CAIR), Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（香港科学院人工智能与机器人中心）；Department of Electrical Engineering, The City University of Hong Kong（香港城市大学电子工程系）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Sixth Affiliated Hospital, Sun Yat-sen University（中山大学第六附属医院）
CPKD: Clinical Prior Knowledge-Constrained Diffusion Models for Surgical Phase Recognition in Endoscopic Submucosal Dissection
School of Biomedical Engineering（生物医学工程学院）；Division of Gastroenterology and Hepatology, Shanghai Institute of Digestive Disease, NHC Key Laboratory of Digestive Diseases, Renji Hospital（消化内科与肝病科、上海消化疾病研究所、国家消化疾病临床医学研究中心、仁济医院）；College of Media Engineering（媒体工程学院）；Aier Institute of Digital Ophthalmology and Visual Science（数字眼科与视觉科学研究院）
Universitat Pompeu Fabra（庞培法布拉大学）；University of Adelaide（阿德莱德大学）；Catalan Institution for Research and Advanced Studies (ICREA)（加泰罗尼亚研究与高级研究所（ICREA））
We study the impact of different loss functions on lesion segmentation from medical images. Although the Cross-Entropy (CE) loss is the most popular option when dealing with natural images, for biomedical image segmentation the soft Dice loss is often preferred due to its ability to handle imbalanced scenarios. On the other hand, the combination of both functions has also been successfully applied in this kind of tasks. A much less studied problem is the generalization ability of all these losses in the presence of Out-of-Distribution (OoD) data. This refers to samples appearing in test time t…
Coronavirus disease 2019 (COVID-19) is a highly contagious virus spreading all around the world. Deep learning has been adopted as an effective technique to aid COVID-19 detection and segmentation from computed tomography (CT) images. The major challenge lies in the inadequate public COVID-19 datasets. Recently, transfer learning has become a widely used technique that leverages the knowledge gained while solving one problem and applying it to a different but related problem. However, it remains unclear whether various non-COVID19 lung lesions could contribute to segmenting COVID-19 infection …
From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM
Zhejiang University（浙江大学）；Shanghai Institute for Advanced Study, Zhejiang University（浙江大学上海高等研究院）；Shanghai Key Laboratory of MICCAI（上海市MICCAI重点实验室）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）；Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University（复旦大学附属中山医院内镜中心及内镜研究所）；Shanghai Collaborative Innovation Center of Endoscopy（上海市内镜协同创新中心）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Qilu Hospital of Shandong University（山东大学齐鲁医院）；National University o（国立大学）
Enhancing Diagnostic Precision in Gastric Bleeding through Automated Lesion Segmentation: A Deep DuS-KFCM Approach
University of Macau（澳门大学）；Guangdong Institute of Intelligence Science and Technology（广东省智能科学与技术研究院）；The Chinese University of Hong Kong（香港中文大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Chongqing Technology and Business University（重庆工商大学）；Institute of Artificial Intelligence, Chongqing Technology and Business University（重庆工商大学人工智能研究院）；Hebei University of Engineering（河北工程大学）
Simple is what you need for efficient and accurate medical image segmentation
Department of Ultrasound, Shenzhen Maternity and Child Healthcare Hospital, Women and Children's Medical Center, Southern Medical University, Shenzhen, Guangdong Province, China（超声科，深圳母婴保健医院，妇女儿童医疗中心，南方医科大学，深圳，广东省，中国）
University College London（伦敦大学学院）；Scuola Superiore Sant’Anna（圣安娜高等研究院）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Toshiba Corporate Research and Development Center（东芝企业研发中心）；The Royal Infirmary of Edinburgh（爱丁堡皇家医院）；University of Turin（都灵大学）
Early diagnosis is essential for the successful treatment of bowel cancers including colorectal cancer (CRC) and capsule endoscopic imaging with robotic actuation can be a valuable diagnostic tool when combined with automated image analysis. We present a deep learning rooted detection and segmentation framework for recognizing lesions in colonoscopy and capsule endoscopy images. We restructure established convolution architectures, such as VGG and ResNets, by converting them into fully-connected convolution networks (FCNs), fine-tune them and study their capabilities for polyp segmentation and…
Shanghai Jiao Tong University（上海交通大学）；Xi’an Jiaotong University（西安交通大学）；The University of Sydney（悉尼大学）；Shanghai Tenth People’s Hospital, Tongji University（同济大学附属上海市第十人民医院）；Changchun GeneScience Pharmaceutical Co. LTD（长春金赛药业股份有限公司）
Contrastive learning, which aims to capture general representation from unlabeled images to initialize the medical analysis models, has been proven effective in alleviating the high demand for expensive annotations. Current methods mainly focus on instance-wise comparisons to learn the global discriminative features, however, pretermitting the local details to distinguish tiny anatomical structures, lesions, and tissues. To address this challenge, in this paper, we propose a general unsupervised representation learning framework, named local discrimination (LD), to learn local discriminative f…
SimulaMet（西姆拉大都会研究所）；Augere Medical AS（奥杰医疗公司）；UiT The Arctic University of Norway（挪威UiT北极大学）；University of Oslo（奥斯陆大学）；Oslo Metropolitan University（奥斯陆城市大学）；Sahlgrenska University Hospital（萨尔格伦斯卡大学医院）；Bærum Hospital（贝鲁姆医院）
Colonoscopy is considered the gold standard for detection of colorectal cancer and its precursors. Existing examination methods are, however, hampered by high overall miss-rate, and many abnormalities are left undetected. Computer-Aided Diagnosis systems based on advanced machine learning algorithms are touted as a game-changer that can identify regions in the colon overlooked by the physicians during endoscopic examinations, and help detect and characterize lesions. In previous work, we have proposed the ResUNet++ architecture and demonstrated that it produces more efficient results compared …
A Client-server Deep Federated Learning for Cross-domain Surgical Image Segmentation
NepAl Applied Mathematics and Informatics Institute (NAAMII)（尼泊尔应用数学与信息学研究所（NAAMII））；University of Leeds（利兹大学）；University of Liverpool（利物浦大学）；University College London（伦敦大学学院）；University of Aberdeen（阿伯丁大学）
SimulaMet；UiT The Arctic University of Norway（挪威北极大学）；Oslo Metropolitan University（奥斯陆城市大学）
Semantic image segmentation is the process of labeling each pixel of an image with its corresponding class. An encoder-decoder based approach, like U-Net and its variants, is a popular strategy for solving medical image segmentation tasks. To improve the performance of U-Net on various segmentation tasks, we propose a novel architecture called DoubleU-Net, which is a combination of two U-Net architectures stacked on top of each other. The first U-Net uses a pre-trained VGG-19 as the encoder, which has already learned features from ImageNet and can be transferred to another task easily. To capt…
SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data
UCL Hawkes Institute, University College London（伦敦大学学院霍克斯学院）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）；Medtronic（美敦力）；Department of Computer Science, University Liverpool（利物浦大学计算机科学系）
Local discriminative representation is needed in many medical image analysis tasks such as identifying sub-types of lesion or segmenting detailed components of anatomical structures. However, the commonly applied supervised representation learning methods require a large amount of annotated data, and unsupervised discriminative representation learning distinguishes different images by learning a global feature, both of which are not suitable for localized medical image analysis tasks. In order to avoid the limitations of these two methods, we introduce local discrimination into unsupervised re…
SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image Segmentation
Shanghai AI Laboratory（上海人工智能实验室）；Xiamen University（厦门大学）；East China Normal University（华东师范大学）；University of Cambridge（剑桥大学）；Stanford University（斯坦福大学）
Brno University of Technology（布尔诺理工大学）；University of Las Palmas de Gran Canaria（大加那利岛拉斯帕尔马斯大学）；Masaryk University（马萨里克大学）；University Hospital Brno（布尔诺大学医院）
We present a novel approach of 2D to 3D transfer learning based on mapping pre-trained 2D convolutional neural network weights into planar 3D kernels. The method is validated by the proposed planar 3D res-u-net network with encoder transferred from the 2D VGG-16, which is applied for a single-stage unbalanced 3D image data segmentation. In particular, we evaluate the method on the MICCAI 2016 MS lesion segmentation challenge dataset utilizing solely fluid-attenuated inversion recovery (FLAIR) sequence without brain extraction for training and inference to simulate real medical praxis. The plan…
The First Affiliated Hospital of Jinan University（暨南大学附属第一医院）；The Fifth Affiliated Hospital of Sun Yat-sen University（中山大学附属第五医院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong（香港中文大学）；Future Network of Intelligence Institute (FNii)（未来智能网络研究院）；Sun Yat-sen University（中山大学）；Jinan University（暨南大学）；Guangdong Provincial Key Laboratory of Big Data Computing（广东省大数据计算重点实验室）
Medical imaging technologies, including computed tomography (CT) or chest X-Ray (CXR), are largely employed to facilitate the diagnosis of the COVID-19. Since manual report writing is usually too time-consuming, a more intelligent auxiliary medical system that could generate medical reports automatically and immediately is urgently needed. In this article, we propose to use the medical visual language BERT (Medical-VLBERT) model to identify the abnormality on the COVID-19 scans and generate the medical report automatically based on the detected lesion regions. To produce more accurate medical …
