TG-LMM: Enhancing Medical Image Segmentation Accuracy through Text-Guided Large Multi-Modal Model
Sun Yat-sen University（中山大学）；National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院国家癌症中心/国家肿瘤临床医学研究中心/肿瘤医院&深圳医院）；University of Macau（澳门大学）
Yihao Zhao, Enhao Zhong, Cuiyun Yuan, Yang Li, Man Zhao, Chunxia Li, Jun Hu, Chenbin Liu
We propose TG-LMM (Text-Guided Large Multi-Modal Model), a novel approach that leverages textual descriptions of organs to enhance segmentation accuracy in medical images. Existing medical image segmentation methods face several challenges: current medical automatic segmentation models do not effectively utilize prior knowledge, such as descriptions of organ locations; previous text-visual models focus on identifying the target rather than improving the segmentation accuracy; prior models attempt to use prior knowledge to enhance accuracy but do not incorporate pre-trained models. To address these issues, TG-LMM integrates prior knowledge, specifically expert descriptions of the spatial locations of organs, into the segmentation process. Our model utilizes pre-trained image and text encoders to reduce the number of training parameters and accelerate the training process. Additionally, we designed a comprehensive image-text information fusion structure to ensure thorough integration of the two modalities of data. We evaluated TG-LMM on three authoritative medical image datasets, encompassing the segmentation of various parts of the human body. Our method demonstrated superior performance compared to existing approaches, such as MedSAM, SAM and nnUnet.
CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation
Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；School of Computer Science, the University of Sydney（悉尼大学计算机学院）；China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司）
Semi-Supervised Multi-Modal Medical Image Segmentation for Complex Situations
School of Physics, Peking University, Beijing, China（北京大学物理系）；School of Computer Science, Peking University, Beijing, China（北京大学计算机系）；Department of Radiotherapy, Peking University Cancer Hospital, Beijing, China（北京大学肿瘤医院放疗科）
DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation
College of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities
Mohamed Bin Zayed University of Artificial Intelligence（Mohamed Bin Zayed大学人工智能学院）；Linköping University（林肯大学）；Shaikh Tahnoon bin Mohammed Medical City（Shaikh Tahnoon bin Mohammed医疗城）；Tawam Hospital（Tawam医院）；Sheikh Shakhbout Medical City（Sheikh Shakhbout医疗城）；Govt Medical College Kozhikode（科钦政府医学院）
Disentangled representation is a powerful technique to tackle domain shift problem in medical image analysis in unsupervised domain adaptation setting.However, previous methods only focus on exacting domain-invariant feature and ignore whether exacted feature is meaningful for downstream tasks.We propose a novel framework, called semantic-guided disentangled representation (SGDR), an effective method to exact semantically meaningful feature for segmentation task to improve performance of cross modality medical image segmentation in unsupervised domain adaptation setting. To exact the meaningfu…
The University of Hong Kong（香港大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of Cambridge（剑桥大学）
Learning medical visual representations directly from paired radiology reports has become an emerging topic in representation learning. However, existing medical image-text joint learning methods are limited by instance or local supervision analysis, ignoring disease-level semantic correspondences. In this paper, we present a novel Multi-Granularity Cross-modal Alignment (MGCA) framework for generalized medical visual representation learning by harnessing the naturally exhibited semantic correspondences between medical image and radiology reports at three different levels, i.e., pathological r…
Université de Rouen Normandie（鲁昂诺曼底大学）；INSA Rouen（鲁昂应用科学学院）
Multi-modality is widely used in medical imaging, because it can provide multiinformation about a target (tumor, organ or tissue). Segmentation using multimodality consists of fusing multi-information to improve the segmentation. Recently, deep learning-based approaches have presented the state-of-the-art performance in image classification, segmentation, object detection and tracking tasks. Due to their self-learning and generalization ability over large amounts of data, deep learning recently has also gained great interest in multi-modal medical image segmentation. In this paper, we give an …
Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
CVIU Lab, University of Arkansas, USA（美国亚拉巴马大学CVIU实验室）；Vietnam National University, Ho Chi Minh City University of Science, Vietnam（越南国家大学胡志明市科技大学）；Carnegie Mellon University, USA（美国卡内基梅隆大学）
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models
School of Data Science, Fudan University（复旦大学数据科学学院）；ByteDance（字节跳动）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Research Institute of Intelligent and Complex Systems, Fudan University（复旦大学智能与复杂系统研究院）；Shanghai Innovation Institute（上海创新研究院）
Multi-modal Auto-regressive Modeling via Visual Words
School of Computer Science, Wuhan University（武汉大学计算机学院）；Shanghai Jiao Tong University（上海交通大学）；School of Information Management, Wuhan University（武汉大学信息管理学院）
RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）
AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation
Fudan University（复旦大学）；Xidian University（西安电子科技大学）；KAIST（韩国科学技术院）；Children’s Hospital of Fudan University（复旦大学附属儿童医院）；Shanghai AI Laboratory（上海人工智能实验室）；Hong Kong Polytechnic University（香港理工大学）
End-to-End Breast Cancer Radiotherapy Planning via LMMs with Consistency Embedding
Samsung Research（三星研究院）；Center for Advanced Medical Computing and Analysis (CAMCA)（先进医学计算与分析中心）；Massachusetts General Hospital (MGH) and Harvard Medical School（麻省总医院和哈佛医学院）；Yonsei University College of Medicine（延世大学医学院）；Yonsei University（延世大学）；Institute for Innovation in Digital Healthcare（数字医疗创新研究所）；Yongin Severance Hospital（Yongin Severance医院）；Gachon University Gil Hospital（高仁大学Gil医院）；Oncosoft Inc（Oncosoft公司）；Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST)（人工智能研究生院，韩国科学技术院）
This paper presents a new approach of transfer learning-based medical image classification to mitigate insufficient labeled data problem in medical domain. Instead of direct transfer learning from source to small number of labeled target data, we propose a modality-bridge transfer learning which employs the bridge database in the same medical imaging acquisition modality as target database. By learning the projection function from source to bridge and from bridge to target, the domain difference between source (e.g., natural images) and target (e.g., X-ray images) can be mitigated. Experimenta…
Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependenci…
Numerous factors could lead to partial deteriorations of medical images. For example, metallic implants will lead to localized perturbations in MRI scans. This will affect further post-processing tasks such as attenuation correction in PET/MRI or radiation therapy planning. In this work, we propose the inpainting of medical images via Generative Adversarial Networks (GANs). The proposed framework incorporates two patch-based discriminator networks with additional style and perceptual losses for the inpainting of missing information in realistically detailed and contextually consistent manner. …
MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZIU-UIUC Institute, Zhejiang University（浙江大学ZIU-UIUC研究院）；School of Public Health, Zhejiang University（浙江大学公共卫生学院）
M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）
Large Multi-modal Model Cartographic Map Comprehension for Textual Locality Georeferencing
School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand（梅西大学数学与计算科学学院）；School of Computer Science and Informatics, Cardiff University, Cardiff, UK（卡迪夫大学计算机科学与信息学院）
