Parameter-Efficient VLMs for Gastrointestinal Endoscopy: Medical Image Generation and Clinical Visual Question Answering
Computer Science Department, Morgan State University（莫尔甘州大学计算机科学系）；International Organization for Migration (IOM)（国际移民组织）；Electrical & Computer Engineering Department, Morgan State University（莫尔甘州大学电气与计算机工程系）
The major limitations of gastrointestinal (GI) endoscopy AI systems arise from a shortage of annotated data, strict privacy policies, and significant bottlenecks in conventional model fine-tuning. Such limitations impede the successful application of sophisticated AI models in clinical practice, particularly affecting the reliability and scalability of diagnosis. In this paper, we present a dual-pipeline PEFT model that addresses two fundamental problems: medical Visual Question Answering (VQA) and the generation of privacy-preserving synthetic data. For clinical VQA, we adopt the Florence-2 vision-language model. Leveraging PEFT enhances model interpretability while substantially reducing the computational cost of training. Simultaneously, we employ Low-Rank Adaptation (LoRA) with Stable Diffusion 2.1 to generate high-quality GI images that enhance training databases without violating patient privacy. This research utilized the Kvasir-VQA dataset. Our Florence-2 VQA model achieved ROUGE-1 of 0.92, ROUGE-L of 0.91, and BLEU score improvements from 0.08 to 0.24. Fine-tuning on private datasets consistently showed better results than fine-tuning on public datasets. The rank-4 LoRA synthesis achieved optimal performance with a fidelity score of 0.290, an agreement score of 0.730, and a Frechet BiomedCLIP Distance (FBD) of 1450, reducing computational costs by almost 90 percent. This framework improves the clinical potential of AI in GI endoscopy. Compared to FLUX, MSDM, and Kandinsky 2.2, our model demonstrates superior FBD and strong semantic alignment. While other models lead in Fidelity or Agreement, our lower FBD indicates better image-text coherence. These results establish our approach as a robust solution for enhancing VQA and synthetic data generation in clinical AI.
GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards
Zhongshan Hospital, Fudan University（复旦大学附属中山医院）；School of Basic Medical Sciences, Fudan University（复旦大学基础医学院）；Microsoft Research Asia（微软亚洲研究院）；Data Science Institute, Imperial College London（帝国理工学院数据科学研究所）
Development and multi-center evaluation of domain-adapted speech recognition for human-AI teaming in real-world gastrointestinal endoscopy
Zhejiang University（浙江大学）；Shanghai Institute for Advanced Study, Zhejiang University（浙江大学上海高等研究院）；Shanghai Key Laboratory of MICCAI（上海市MICCAI重点实验室）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）；Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University（复旦大学附属中山医院内镜中心及内镜研究所）；Shanghai Collaborative Innovation Center of Endoscopy（上海市内镜协同创新中心）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy
Simula Metropolitan Center for Digital Engineering (SimulaMet)（Simula Metropolitan Center for Digital Engineering）；Oslo Metropolitan University (OsloMet)（Oslo Metropolitan University）；Simula Research Laboratory
Medico 2025: Visual Question Answering for Gastrointestinal Imaging
SimulaMet - Simula Metropolitan Center for Digital Engineering, Oslo, Norway（SimulaMet - Simula Metropolitan Center for Digital Engineering，挪威奥斯陆）；Simula Research Laboratory, Oslo, Norway（Simula研究实验室，挪威奥斯陆）；OsloMet - Oslo Metropolitan University, Oslo, Norway（OsloMet - 奥斯陆 Metropolitan 大学，挪威奥斯陆）
This paper attempts to provide the reader a place to begin studying the application of computer vision and machine learning to gastrointestinal (GI) endoscopy. They have been classified into 18 categories. It should be be noted by the reader that this is a review from pre-deep learning era. A lot of deep learning based applications have not been covered in this thesis.
On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos
Indian Institute of Technology Delhi（印度理工学院德里分校）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；Birmingham City University（伯明翰城市大学）；University Hospitals Birmingham（伯明翰大学医院）；Khalifa University（哈利法大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Models
Rochester Institute of Technology, Rochester, NY, USA（罗切斯特技术学院）；Nepal Applied Mathematics and Informatics Institute for Research (NAAMII)（尼泊尔应用数学与信息技术研究所）；Kathmandu University（加德满都大学）；University of Lausanne（洛桑大学）；University of Aberdeen（阿伯丁大学）
SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis
Nepal Applied Mathematics and Informatics Institute for Research（尼泊尔应用数学与信息技术研究所）；GastroIntestinal Department, Dhulikhel Hospital（杜尔基hel医院消化内科）；Univesity of Lausanne（洛桑大学）；University of West Virginia（西弗吉尼亚大学）；University of Utah（犹他大学）；University of Aberdeen（阿伯丁大学）
Imperial College London（伦敦帝国学院）；Ulm University（乌尔姆大学）
For early diagnosis of malignancies in the gastrointestinal tract, surveillance endoscopy is increasingly used to monitor abnormal tissue changes in serial examinations of the same patient. Despite successes with optical biopsy for in vivo and in situ tissue characterisation, biopsy retargeting for serial examinations is challenging because tissue may change in appearance between examinations. In this paper, we propose an inter-examination retargeting framework for optical biopsy, based on an image descriptor designed for matching between endoscopic scenes over significant time intervals. Each…
National University of Singapore（新加坡国立大学）；National University of Singapore Suzhou Research Institute（新加坡国立大学苏州研究院）
Domain Generalization is a challenging topic in computer vision, especially in Gastrointestinal Endoscopy image analysis. Due to several device limitations and ethical reasons, current open-source datasets are typically collected on a limited number of patients using the same brand of sensors. Different brands of devices and individual differences will significantly affect the model's generalizability. Therefore, to address the generalization problem in GI(Gastrointestinal) endoscopy, we propose a multi-domain GI dataset and a light, plug-in block called InvNorm(Invertible Normalization), whic…
PDPM Indian Institute of Information Technology, Design and Manufacturing（PDPM印度信息技术设计与制造学院）
Wireless Capsule Endoscopy is one of the most advanced non-invasive methods for the examination of gastrointestinal tracts. An intelligent computer-aided diagnostic system for detecting gastrointestinal abnormalities like polyp, bleeding, inflammation, etc. is highly exigent in wireless capsule endoscopy image analysis. Abnormalities greatly differ in their shape, size, color, and texture, and some appear to be visually similar to normal regions. This poses a challenge in designing a binary classifier due to intra-class variations. In this study, a hybrid convolutional neural network is propos…
University of Oxford（牛津大学）；Oxford NIHR Biomedical Research Centre（牛津NIHR生物医学研究中心）；Arab Academy for Science and Technology（阿拉伯科学技术学院）；University College London（伦敦大学学院）；Middle East Technical University（中东技术大学）；University of Würzburg（维尔茨堡大学）；University of Central Lancashire（中央兰开夏大学）；Ibn Zohr University（伊本·佐赫尔大学）
The Endoscopy Computer Vision Challenge (EndoCV) is a crowd-sourcing initiative to address eminent problems in developing reliable computer aided detection and diagnosis endoscopy systems and suggest a pathway for clinical translation of technologies. Whilst endoscopy is a widely used diagnostic and treatment tool for hollow-organs, there are several core challenges often faced by endoscopists, mainly: 1) presence of multi-class artefacts that hinder their visual interpretation, and 2) difficulty in identifying subtle precancerous precursors and cancer abnormalities. Artefacts often affect the…
Multi-contrast laser endoscopy for in vivo gastrointestinal imaging
Department of Biomedical Engineering（生物医学工程系）；Johns Hopkins University（约翰霍普金斯大学）；Division of Gastroenterology and Hepatology（消化内科与肝病科）；Johns Hopkins Hospital（约翰霍普金斯医院）
The article presents a new multi-label comprehensive image dataset from flexible endoscopy, colonoscopy and capsule endoscopy, named ERS. The collection has been labeled according to the full medical specification of 'Minimum Standard Terminology 3.0' (MST 3.0), describing all possible findings in the gastrointestinal tract (104 possible labels), extended with an additional 19 labels useful in common machine learning applications. The dataset contains around 6000 precisely and 115,000 approximately labeled frames from endoscopy videos, 3600 precise and 22,600 approximate segmentation masks, an…
Exploring Self-Supervised Learning with U-Net Masked Autoencoders and EfficientNet-B7 for Improved Gastrointestinal Abnormality Classification in Video Capsule Endoscopy
International Institute of Information Technology, Bangalore（国际信息技术学院，班加罗尔）；Indian Institute of Technology, Hyderabad（印度理工学院，海得拉巴）
MultiAttenGastro: Multi-Dimensional Attention Augmentation for Gastrointestinal Endoscopy Classification
Sardar Vallabhbhai National Institute of Technology（萨达尔·瓦拉巴伊·国家理工学院）；Swasthyam Gastro and Liver Hospital（斯瓦斯蒂亚姆胃肠肝病医院）；Norwegian University of Science and Technology（挪威科技大学）
Brac University（布拉克大学）；University of Rochester（罗切斯特大学）；Shaheed Tajuddin Ahmad Medical College（沙希德·塔朱丁·艾哈迈德医学院）
Accurate disease categorization using endoscopic images is a significant problem in Gastroenterology. This paper describes a technique for assisting medical diagnosis procedures and identifying gastrointestinal tract disorders based on the categorization of characteristics taken from endoscopic pictures using a vision transformer and transfer learning model. Vision transformer has shown very promising results on difficult image classification tasks. In this paper, we have suggested a vision transformer based approach to detect gastrointestianl diseases from wireless capsule endoscopy (WCE) cur…
GastroVision: A Multi-class Endoscopy Image Dataset for Computer Aided Gastrointestinal Disease Detection
Northwestern University（西北大学）；Indian Institute of Technology Guwahati（印度理工学院古瓦哈提分校）；Jefferson Health NJ（新泽西杰斐逊医疗）；SimulaMet（西穆拉梅特研究所）；Sahlgrenska University Hospital（萨尔格伦斯卡大学医院）
Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Models
Simula Metropolitan Center for Digital Engineering (SimulaMet), Norway（Simula数字工程中心（SimulaMet））；Oslo Metropolitan University (OsloMet), Norway（奥斯陆 Metropolitan 大学（OsloMet））；Simula Research Laboratory, Norway（Simula研究实验室）
SimulaMet；UIT The Arctic University of Norway（挪威北极大学）；University of Oxford（牛津大学）；Simula Research Laboratory（Simula研究实验室）；Oslo Metropolitan University（奥斯陆城市大学）；Sintef Digital（SINTEF数字研究所）；Augere Medical AS（Augere Medical公司）；Sahlgrenska University Hospital-Mölndal（萨尔格伦斯卡大学医院-默恩达尔）；Bærum Hospital（贝鲁姆医院）；Karolinska University Hospital（卡罗林斯卡大学医院）
Gastrointestinal (GI) pathologies are periodically screened, biopsied, and resected using surgical tools. Usually the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development and amount and size of the resected area get lost. This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic we have released the ``Kvasir-Instrument'' dataset which consists of $590$ annotated frames cont…
Automated Bleeding Detection and Classification in Wireless Capsule Endoscopy with YOLOv8-X
Indian Institute of Technology, Indore (IITI)（印度理工学院印多尔分校）；Thapar Institute of Engineering & Technology (TIET)（塔帕尔工程技术学院）；Vellore Institute of Technology, Chennai (VIT)（韦洛尔理工学院 Chennai 校区）
Video capsule endoscopy (VCE) is used widely nowadays for visualizing the gastrointestinal (GI) tract. Capsule endoscopy exams are prescribed usually as an additional monitoring mechanism and can help in identifying polyps, bleeding, etc. To analyze the large scale video data produced by VCE exams automatic image processing, computer vision, and learning algorithms are required. Recently, automatic polyp detection algorithms have been proposed with various degrees of success. Though polyp detection in colonoscopy and other traditional endoscopy procedure based images is becoming a mature field…
Isfahan University of Technology（伊斯法罕理工大学）；University of Queensland（昆士兰大学）；University of Michigan（密歇根大学）
Recent advances in capsule endoscopy systems have introduced new methods and capabilities. The capsule endoscopy system, by observing the entire digestive tract, has significantly improved diagnosing gastrointestinal disorders and diseases. The system has challenges such as the need to enhance the quality of the transmitted images, low frame rates of transmission, and battery lifetime that need to be addressed. One of the important parts of a capsule endoscopy system is the image compression unit. Better compression of images increases the frame rate and hence improves the diagnosis process. I…
University of Saskatchewan（萨斯喀彻温大学）；Saskatchewan Health Authority（萨斯喀彻温卫生局）
Conventional Endoscopy (CE) and Wireless Capsule Endoscopy (WCE) are known tools for diagnosing gastrointestinal (GI) tract disorders. Detecting the anatomical location of GI tract can help clinicians to determine a more appropriate treatment plan, can reduce repetitive endoscopy and is important in drug-delivery. There are few research that address detecting anatomical location of WCE and CE images using classification, mainly because of difficulty in collecting data and anotating them. In this study, we present a few-shot learning method based on distance metric learning which combines trans…
Capsule Endoscopy Multi-classification via Gated Attention and Wavelet Transformations
National Institute of Technology Puducherry（本地治里国家技术学院）；Sardar Vallabhbhai National Institute of Technology Surat（苏拉特萨达尔·瓦拉巴伊国家技术学院）；Norwegian University of Science and Technology（挪威科技大学）
Isfahan University of Technology（伊斯法罕理工大学）；University of Michigan（密歇根大学）
Wireless capsule endoscopy (WCE) is an effective means of diagnosis of gastrointestinal disorders. Detection of informative scenes by WCE could reduce the length of transmitted videos and can help with the diagnosis. In this paper we propose a simple and efficient method for segmentation of the bleeding regions in WCE captured images. Suitable color channels are selected and classified by a multi-layer perceptron (MLP) structure. The MLP structure is quantized such that the implementation does not require multiplications. The proposed method is tested by simulation on WCE bleeding image datase…
University of Saskatchewan（萨斯喀彻温大学）；Saskatchewan Health Authority（萨斯喀彻温卫生局）
Wireless Capsule Endoscopy (WCE) helps physicians examine the gastrointestinal (GI) tract noninvasively. There are few studies that address pathological assessment of endoscopy images in multiclass classification and most of them are based on binary anomaly detection or aim to detect a specific type of anomaly. Multiclass anomaly detection is challenging, especially when the dataset is poorly sampled or imbalanced. Many available datasets in endoscopy field, such as KID2, suffer from an imbalance issue, which makes it difficult to train a high-performance model. Additionally, increasing the nu…
Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. While the endoscopy video contains a wealth of information, tools to capture this information for the purpose of clinical reporting are rather poor. In date, endoscopists do not have any access to tools that enable them to browse the video data in an efficient and user friendly manner. Fast and reliable video retrieval methods could for example, allow them to review data from previous exams and therefore improve their ability to monitor disease progression. Deep learning provides new ave…
One-shot synthesis of rare gastrointestinal lesions improves diagnostic accuracy and clinical training
Zhejiang University（浙江大学）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海研究院）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学部数字医学研究中心）；Shanghai Collaborative Innovation Center of Endoscopy（上海内镜协同创新中心）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学曼彻斯特商学院）；Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学学院）
Medical endoscopy remains a challenging application for simultaneous localization and mapping (SLAM) due to the sparsity of image features and size constraints that prevent direct depth-sensing. We present a SLAM approach that incorporates depth predictions made by an adversarially-trained convolutional neural network (CNN) applied to monocular endoscopy images. The depth network is trained with synthetic images of a simple colon model, and then fine-tuned with domain-randomized, photorealistic images rendered from computed tomography measurements of human colons. Each image is paired with an …
Seeing More with Less: Video Capsule Endoscopy with Multi-Task Learning
Department of Computer Science, University of Tübingen, Tübingen, Germany（图宾根大学计算机科学系）；Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany（德累斯顿技术大学数字健康研究中心）；Department of Medicine I, University Hospital Dresden, TU Dresden, Dresden, Germany（德累斯顿大学医院第一医学系）
Automating the analysis of imagery of the Gastrointestinal (GI) tract captured during endoscopy procedures has substantial potential benefits for patients, as it can provide diagnostic support to medical practitioners and reduce mistakes via human error. To further the development of such methods, we propose a two-stream model for endoscopic image analysis. Our model fuses two streams of deep feature inputs by mapping their inherent relations through a novel relational network model, to better model symptoms and classify the image. In contrast to handcrafted feature-based models, our proposed …
University of Oxford（牛津大学）；CRO Centro Riferimento Oncologico IRCCS Aviano（阿维亚诺肿瘤参考中心IRCCS）；University of Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；University of Lincoln（林肯大学）
Whilst many technologies are built around endoscopy, there is a need to have a comprehensive dataset collected from multiple centers to address the generalization issues with most deep learning frameworks. What could be more important than disease detection and localization? Through our extensive network of clinical and computational experts, we have collected, curated and annotated gastrointestinal endoscopy video frames. We have released this dataset and have launched disease detection and segmentation challenge EDD2020 https://edd2020.grand-challenge.org to address the limitations and explo…
Scaling Video Pretraining for Surgical Foundation Models
The Johns Hopkins University（约翰霍普金斯大学）；Zhejiang University（浙江大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳-香槟分校联合学院）；Zhejiang Lab（之江实验室）；Shaoxing University（绍兴大学）；Angelalign（时代天使）；Harvard Medical School（哈佛医学院）
EndoOOD: Uncertainty-aware Out-of-distribution Detection in Capsule Endoscopy Diagnosis
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）
University of Oxford（牛津大学）；Imperial College London（帝国理工学院）
Endoscopy is a widely used imaging modality to diagnose and treat diseases in hollow organs as for example the gastrointestinal tract, the kidney and the liver. However, due to varied modalities and use of different imaging protocols at various clinical centers impose significant challenges when generalising deep learning models. Moreover, the assembly of large datasets from different clinical centers can introduce a huge label bias that renders any learnt model unusable. Also, when using new modality or presence of images with rare patterns, a bulk amount of similar image data and their corre…
Semantic Map Guided Synthesis of Wireless Capsule Endoscopy Images using Diffusion Models
Captos Co., Ltd.（Captos株式会社）；Seoreu Co., Ltd.（Seoreu株式会社）；Pusan National University School of Medicine（釜山大学医学院）；Pusan National University Yangsan Hospital（釜山大学梁山医院）；Kyungpook National University（庆北大学）
CAVE-Net: Classifying Abnormalities in Video Capsule Endoscopy
Netaji Subhas University of Technology（贾瓦哈拉尔·尼赫鲁技术大学）；Helios Tech Solutions Pvt Ltd（赫利奥斯技术解决方案私人有限公司）；Centre for Artificial Intelligence and Machine Learning（人工智能与机器学习中心）
