For early diagnosis of malignancies in the gastrointestinal tract, surveillance endoscopy is increasingly used to monitor abnormal tissue changes in serial examinations of the same patient. Despite successes with optical biopsy for in vivo and in situ tissue characterisation, biopsy retargeting for serial examinations is challenging because tissue may change in appearance between examinations. In this paper, we propose an inter-examination retargeting framework for optical biopsy, based on an image descriptor designed for matching between endoscopic scenes over significant time intervals. Each scene is described by a hierarchy of regional intensity comparisons at various scales, offering tolerance to long-term change in tissue appearance whilst remaining discriminative. Binary coding is then used to compress the descriptor via a novel random forests approach, providing fast comparisons in Hamming space and real-time retargeting. Extensive validation conducted on 13 in vivo gastrointestinal videos, collected from six patients, show that our approach outperforms state-of-the-art methods.
This paper attempts to provide the reader a place to begin studying the application of computer vision and machine learning to gastrointestinal (GI) endoscopy. They have been classified into 18 categories. It should be be noted by the reader that this is a review from pre-deep learning era. A lot of deep learning based applications have not been covered in this thesis.
National University of Singapore（新加坡国立大学）；National University of Singapore Suzhou Research Institute（新加坡国立大学苏州研究院）
Domain Generalization is a challenging topic in computer vision, especially in Gastrointestinal Endoscopy image analysis. Due to several device limitations and ethical reasons, current open-source datasets are typically collected on a limited number of patients using the same brand of sensors. Different brands of devices and individual differences will significantly affect the model's generalizability. Therefore, to address the generalization problem in GI(Gastrointestinal) endoscopy, we propose a multi-domain GI dataset and a light, plug-in block called InvNorm(Invertible Normalization), whic…
The article presents a new multi-label comprehensive image dataset from flexible endoscopy, colonoscopy and capsule endoscopy, named ERS. The collection has been labeled according to the full medical specification of 'Minimum Standard Terminology 3.0' (MST 3.0), describing all possible findings in the gastrointestinal tract (104 possible labels), extended with an additional 19 labels useful in common machine learning applications. The dataset contains around 6000 precisely and 115,000 approximately labeled frames from endoscopy videos, 3600 precise and 22,600 approximate segmentation masks, an…
Exploring Self-Supervised Learning with U-Net Masked Autoencoders and EfficientNet-B7 for Improved Gastrointestinal Abnormality Classification in Video Capsule Endoscopy
International Institute of Information Technology, Bangalore（国际信息技术学院，班加罗尔）；Indian Institute of Technology, Hyderabad（印度理工学院，海得拉巴）
MultiAttenGastro: Multi-Dimensional Attention Augmentation for Gastrointestinal Endoscopy Classification
Sardar Vallabhbhai National Institute of Technology（萨达尔·瓦拉巴伊·国家理工学院）；Swasthyam Gastro and Liver Hospital（斯瓦斯蒂亚姆胃肠肝病医院）；Norwegian University of Science and Technology（挪威科技大学）
Multi-contrast laser endoscopy for in vivo gastrointestinal imaging
Department of Biomedical Engineering（生物医学工程系）；Johns Hopkins University（约翰霍普金斯大学）；Division of Gastroenterology and Hepatology（消化内科与肝病科）；Johns Hopkins Hospital（约翰霍普金斯医院）
GastroVision: A Multi-class Endoscopy Image Dataset for Computer Aided Gastrointestinal Disease Detection
Northwestern University（西北大学）；Indian Institute of Technology Guwahati（印度理工学院古瓦哈提分校）；Jefferson Health NJ（新泽西杰斐逊医疗）；SimulaMet（西穆拉梅特研究所）；Sahlgrenska University Hospital（萨尔格伦斯卡大学医院）
Video capsule endoscopy (VCE) is used widely nowadays for visualizing the gastrointestinal (GI) tract. Capsule endoscopy exams are prescribed usually as an additional monitoring mechanism and can help in identifying polyps, bleeding, etc. To analyze the large scale video data produced by VCE exams automatic image processing, computer vision, and learning algorithms are required. Recently, automatic polyp detection algorithms have been proposed with various degrees of success. Though polyp detection in colonoscopy and other traditional endoscopy procedure based images is becoming a mature field…
Isfahan University of Technology（伊斯法罕理工大学）；University of Queensland（昆士兰大学）；University of Michigan（密歇根大学）
Recent advances in capsule endoscopy systems have introduced new methods and capabilities. The capsule endoscopy system, by observing the entire digestive tract, has significantly improved diagnosing gastrointestinal disorders and diseases. The system has challenges such as the need to enhance the quality of the transmitted images, low frame rates of transmission, and battery lifetime that need to be addressed. One of the important parts of a capsule endoscopy system is the image compression unit. Better compression of images increases the frame rate and hence improves the diagnosis process. I…
PDPM Indian Institute of Information Technology, Design and Manufacturing（PDPM印度信息技术设计与制造学院）
Wireless Capsule Endoscopy is one of the most advanced non-invasive methods for the examination of gastrointestinal tracts. An intelligent computer-aided diagnostic system for detecting gastrointestinal abnormalities like polyp, bleeding, inflammation, etc. is highly exigent in wireless capsule endoscopy image analysis. Abnormalities greatly differ in their shape, size, color, and texture, and some appear to be visually similar to normal regions. This poses a challenge in designing a binary classifier due to intra-class variations. In this study, a hybrid convolutional neural network is propos…
Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy
Simula Metropolitan Center for Digital Engineering (SimulaMet)（Simula Metropolitan Center for Digital Engineering）；Oslo Metropolitan University (OsloMet)（Oslo Metropolitan University）；Simula Research Laboratory
University of Oxford（牛津大学）；Oxford NIHR Biomedical Research Centre（牛津NIHR生物医学研究中心）；Arab Academy for Science and Technology（阿拉伯科学技术学院）；University College London（伦敦大学学院）；Middle East Technical University（中东技术大学）；University of Würzburg（维尔茨堡大学）；University of Central Lancashire（中央兰开夏大学）；Ibn Zohr University（伊本·佐赫尔大学）
The Endoscopy Computer Vision Challenge (EndoCV) is a crowd-sourcing initiative to address eminent problems in developing reliable computer aided detection and diagnosis endoscopy systems and suggest a pathway for clinical translation of technologies. Whilst endoscopy is a widely used diagnostic and treatment tool for hollow-organs, there are several core challenges often faced by endoscopists, mainly: 1) presence of multi-class artefacts that hinder their visual interpretation, and 2) difficulty in identifying subtle precancerous precursors and cancer abnormalities. Artefacts often affect the…
Isfahan University of Technology（伊斯法罕理工大学）；University of Michigan（密歇根大学）
Wireless capsule endoscopy (WCE) is an effective means of diagnosis of gastrointestinal disorders. Detection of informative scenes by WCE could reduce the length of transmitted videos and can help with the diagnosis. In this paper we propose a simple and efficient method for segmentation of the bleeding regions in WCE captured images. Suitable color channels are selected and classified by a multi-layer perceptron (MLP) structure. The MLP structure is quantized such that the implementation does not require multiplications. The proposed method is tested by simulation on WCE bleeding image datase…
Parameter-Efficient VLMs for Gastrointestinal Endoscopy: Medical Image Generation and Clinical Visual Question Answering
Computer Science Department, Morgan State University（莫尔甘州大学计算机科学系）；International Organization for Migration (IOM)（国际移民组织）；Electrical & Computer Engineering Department, Morgan State University（莫尔甘州大学电气与计算机工程系）
GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards
Zhongshan Hospital, Fudan University（复旦大学附属中山医院）；School of Basic Medical Sciences, Fudan University（复旦大学基础医学院）；Microsoft Research Asia（微软亚洲研究院）；Data Science Institute, Imperial College London（帝国理工学院数据科学研究所）
SimulaMet；UIT The Arctic University of Norway（挪威北极大学）；University of Oxford（牛津大学）；Simula Research Laboratory（Simula研究实验室）；Oslo Metropolitan University（奥斯陆城市大学）；Sintef Digital（SINTEF数字研究所）；Augere Medical AS（Augere Medical公司）；Sahlgrenska University Hospital-Mölndal（萨尔格伦斯卡大学医院-默恩达尔）；Bærum Hospital（贝鲁姆医院）；Karolinska University Hospital（卡罗林斯卡大学医院）
Gastrointestinal (GI) pathologies are periodically screened, biopsied, and resected using surgical tools. Usually the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development and amount and size of the resected area get lost. This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic we have released the ``Kvasir-Instrument'' dataset which consists of $590$ annotated frames cont…
Seeing More with Less: Video Capsule Endoscopy with Multi-Task Learning
Department of Computer Science, University of Tübingen, Tübingen, Germany（图宾根大学计算机科学系）；Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany（德累斯顿技术大学数字健康研究中心）；Department of Medicine I, University Hospital Dresden, TU Dresden, Dresden, Germany（德累斯顿大学医院第一医学系）
Capsule Endoscopy Multi-classification via Gated Attention and Wavelet Transformations
National Institute of Technology Puducherry（本地治里国家技术学院）；Sardar Vallabhbhai National Institute of Technology Surat（苏拉特萨达尔·瓦拉巴伊国家技术学院）；Norwegian University of Science and Technology（挪威科技大学）
Brac University（布拉克大学）；University of Rochester（罗切斯特大学）；Shaheed Tajuddin Ahmad Medical College（沙希德·塔朱丁·艾哈迈德医学院）
Accurate disease categorization using endoscopic images is a significant problem in Gastroenterology. This paper describes a technique for assisting medical diagnosis procedures and identifying gastrointestinal tract disorders based on the categorization of characteristics taken from endoscopic pictures using a vision transformer and transfer learning model. Vision transformer has shown very promising results on difficult image classification tasks. In this paper, we have suggested a vision transformer based approach to detect gastrointestianl diseases from wireless capsule endoscopy (WCE) cur…
Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. While the endoscopy video contains a wealth of information, tools to capture this information for the purpose of clinical reporting are rather poor. In date, endoscopists do not have any access to tools that enable them to browse the video data in an efficient and user friendly manner. Fast and reliable video retrieval methods could for example, allow them to review data from previous exams and therefore improve their ability to monitor disease progression. Deep learning provides new ave…
CAVE-Net: Classifying Abnormalities in Video Capsule Endoscopy
Netaji Subhas University of Technology（贾瓦哈拉尔·尼赫鲁技术大学）；Helios Tech Solutions Pvt Ltd（赫利奥斯技术解决方案私人有限公司）；Centre for Artificial Intelligence and Machine Learning（人工智能与机器学习中心）
Automating the analysis of imagery of the Gastrointestinal (GI) tract captured during endoscopy procedures has substantial potential benefits for patients, as it can provide diagnostic support to medical practitioners and reduce mistakes via human error. To further the development of such methods, we propose a two-stream model for endoscopic image analysis. Our model fuses two streams of deep feature inputs by mapping their inherent relations through a novel relational network model, to better model symptoms and classify the image. In contrast to handcrafted feature-based models, our proposed …
Medical endoscopy remains a challenging application for simultaneous localization and mapping (SLAM) due to the sparsity of image features and size constraints that prevent direct depth-sensing. We present a SLAM approach that incorporates depth predictions made by an adversarially-trained convolutional neural network (CNN) applied to monocular endoscopy images. The depth network is trained with synthetic images of a simple colon model, and then fine-tuned with domain-randomized, photorealistic images rendered from computed tomography measurements of human colons. Each image is paired with an …
Effective and rapid detection of lesions in the Gastrointestinal tract is critical to gastroenterologist's response to some life-threatening diseases. Wireless Capsule Endoscopy (WCE) has revolutionized traditional endoscopy procedure by allowing gastroenterologists visualize the entire GI tract non-invasively. Once the tiny capsule is swallowed, it sequentially capture images of the GI tract at about 2 to 6 frames per second (fps). A single video can last up to 8 hours producing between 30,000 to 100,000 images. Automating the detection of frames containing specific lesion in WCE video would …
Physicians use Capsule Endoscopy (CE) as a non-invasive and non-surgical procedure to examine the entire gastrointestinal (GI) tract for diseases and abnormalities. A single CE examination could last between 8 to 11 hours generating up to 80,000 frames which is compiled as a video. Physicians have to review and analyze the entire video to identify abnormalities or diseases before making diagnosis. This review task can be very tedious, time consuming and prone to error. While only as little as a single frame may capture useful content that is relevant to the physicians' final diagnosis, frames …
Gastrointestinal Mucosal Problems Classification with Deep Learning
School of Medicine, Islamic Azad University Medical Sciences Najaf Abad（伊斯兰阿扎德大学纳贾夫阿巴德医学科学校区医学院）；School of Medicine, Isfahan University of Medical Sciences（伊斯法罕医科大学医学院）；Department of Computer Engineering, Science and Research Branch, Islamic Azad University（伊斯兰阿扎德大学科学研究校区计算机工程系）
Automated Bleeding Detection and Classification in Wireless Capsule Endoscopy with YOLOv8-X
Indian Institute of Technology, Indore (IITI)（印度理工学院印多尔分校）；Thapar Institute of Engineering & Technology (TIET)（塔帕尔工程技术学院）；Vellore Institute of Technology, Chennai (VIT)（韦洛尔理工学院 Chennai 校区）
University of Saskatchewan（萨斯喀彻温大学）；Saskatchewan Health Authority（萨斯喀彻温卫生局）
Conventional Endoscopy (CE) and Wireless Capsule Endoscopy (WCE) are known tools for diagnosing gastrointestinal (GI) tract disorders. Detecting the anatomical location of GI tract can help clinicians to determine a more appropriate treatment plan, can reduce repetitive endoscopy and is important in drug-delivery. There are few research that address detecting anatomical location of WCE and CE images using classification, mainly because of difficulty in collecting data and anotating them. In this study, we present a few-shot learning method based on distance metric learning which combines trans…
University of Oxford（牛津大学）；Imperial College London（帝国理工学院）
Endoscopy is a widely used imaging modality to diagnose and treat diseases in hollow organs as for example the gastrointestinal tract, the kidney and the liver. However, due to varied modalities and use of different imaging protocols at various clinical centers impose significant challenges when generalising deep learning models. Moreover, the assembly of large datasets from different clinical centers can introduce a huge label bias that renders any learnt model unusable. Also, when using new modality or presence of images with rare patterns, a bulk amount of similar image data and their corre…
Medico 2025: Visual Question Answering for Gastrointestinal Imaging
SimulaMet - Simula Metropolitan Center for Digital Engineering, Oslo, Norway（SimulaMet - Simula Metropolitan Center for Digital Engineering，挪威奥斯陆）；Simula Research Laboratory, Oslo, Norway（Simula研究实验室，挪威奥斯陆）；OsloMet - Oslo Metropolitan University, Oslo, Norway（OsloMet - 奥斯陆 Metropolitan 大学，挪威奥斯陆）
Semantic Map Guided Synthesis of Wireless Capsule Endoscopy Images using Diffusion Models
Captos Co., Ltd.（Captos株式会社）；Seoreu Co., Ltd.（Seoreu株式会社）；Pusan National University School of Medicine（釜山大学医学院）；Pusan National University Yangsan Hospital（釜山大学梁山医院）；Kyungpook National University（庆北大学）
SimulaMet；UiT The Arctic University of Norway（挪威北极大学）；Oslo Metropolitan University（奥斯陆城市大学）；Institute of Biomedical Engineering, University of Oxford（牛津大学生物医学工程研究所）；Bærum Hospital（贝鲁姆医院）；Augere Medical AS（Augere Medical 公司）；Sahlgrenska University Hospital-Mölndal Hospital（萨尔格伦斯卡大学医院-默恩达尔医院）；Sahlgrenska Academy, University of Gothenburg（哥德堡大学萨尔格伦斯卡学院）
Deep learning in gastrointestinal endoscopy can assist to improve clinical performance and be helpful to assess lesions more accurately. To this extent, semantic segmentation methods that can perform automated real-time delineation of a region-of-interest, e.g., boundary identification of cancer or precancerous lesions, can benefit both diagnosis and interventions. However, accurate and real-time segmentation of endoscopic images is extremely challenging due to its high operator dependence and high-definition image quality. To utilize automated methods in clinical settings, it is crucial to de…
University of Saskatchewan（萨斯喀彻温大学）；Saskatchewan Health Authority（萨斯喀彻温卫生局）
Wireless Capsule Endoscopy (WCE) helps physicians examine the gastrointestinal (GI) tract noninvasively. There are few studies that address pathological assessment of endoscopy images in multiclass classification and most of them are based on binary anomaly detection or aim to detect a specific type of anomaly. Multiclass anomaly detection is challenging, especially when the dataset is poorly sampled or imbalanced. Many available datasets in endoscopy field, such as KID2, suffer from an imbalance issue, which makes it difficult to train a high-performance model. Additionally, increasing the nu…
One-shot synthesis of rare gastrointestinal lesions improves diagnostic accuracy and clinical training
Zhejiang University（浙江大学）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海研究院）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学部数字医学研究中心）；Shanghai Collaborative Innovation Center of Endoscopy（上海内镜协同创新中心）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学曼彻斯特商学院）；Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学学院）
Institute for Medical Engineering and Science（医学工程与科学研究所）；Massachusetts Institute of Technology（麻省理工学院）；Lyft Inc.（来福特公司）；Neuromation OU（神经自动化有限公司）；University of Michigan（密歇根大学）
Accurate detection and localization for angiodysplasia lesions is an important problem in early stage diagnostics of gastrointestinal bleeding and anemia. Gold-standard for angiodysplasia detection and localization is performed using wireless capsule endoscopy. This pill-like device is able to produce thousand of high enough resolution images during one passage through gastrointestinal tract. In this paper we present our winning solution for MICCAI 2017 Endoscopic Vision SubChallenge: Angiodysplasia Detection and Localization its further improvements over the state-of-the-art results using sev…
South Central University for Nationalities（中南民族大学）；Nanjing Eastern Theater General Hospital（南京东部战区总医院）
Wireless Capsule Endoscopy(WCE) has been widely used for the screening of gastrointestinal(GI) diseases, especially the small intestine, due to its advantages of non-invasive and painless imaging of the entire digestive tract.However, the huge amount of image data captured by WCE makes manual reading a process that requires a huge amount of tasks and can easily lead to missed detection and false detection of lesions.Therefore, In this paper, we propose a \textbf{T}wo-stage \textbf{M}ulti-scale \textbf{F}eature-fusion learning network(\textbf{TMFNet}) to automatically detect small intestinal po…
Foundational Models for Pathology and Endoscopy Images: Application for Gastric Inflammation
Imperial College London（帝国理工学院）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；University of Latvia（拉脱维亚大学）；Instituto de Engenharia de Sistemas e Computadores, Tecnologia e Ciência（系统与计算机工程、技术与科学研究所）；Universidade do Porto（波尔图大学）；StratejAI（斯特拉泰人工智能公司）；Portuguese Oncology Institute of Porto (IPO Porto)（波尔图葡萄牙肿瘤研究所）；Instituto Investigación Sanitaria INCLIVA（INCLIVA卫生研究所）；Hospital Clínico Universitario de Valencia（瓦伦西亚大学临床医院）；Yale University（耶鲁大学）
A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI
Department of Computer Science and Engineering（计算机科学与工程系）
Federated Foundation Model for GI Endoscopy Images
Lane Department of Computer Science and Electrical Engineering, West Virginia University（沃里奇大学计算机科学与电气工程系）；West Virginia University（沃里奇大学）；School of Medicine, West Virginia University（沃里奇大学医学院）；University of Memphis（密苏里州孟菲斯大学）；University of Aberdeen（阿伯丁大学）
