Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning
Sydney AI Centre, The University of Sydney（悉尼大学悉尼人工智能中心）；Institute of Image Communication and Network Engineering, Shanghai JiaoTong University（上海交通大学图像通信与网络工程研究所）
Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, Tongliang Liu
Although vision models such as Contrastive Language-Image Pre-Training (CLIP) show impressive generalization performance, their zero-shot robustness is still limited under Out-of-Distribution (OOD) scenarios without fine-tuning. Instead of undesirably providing human supervision as commonly done, it is possible to take advantage of Multi-modal Large Language Models (MLLMs) that hold powerful visual understanding abilities. However, MLLMs are shown to struggle with vision problems due to the incompatibility of tasks, thus hindering their utilization. In this paper, we propose to effectively leverage MLLMs to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner. To solve the incompatibility issue, we propose a novel Denoising In-Context Learning (DICL) strategy to align vision tasks with MLLMs. Concretely, by estimating a transition matrix that captures the probability of one class being confused with another, an instruction containing a correct exemplar and an erroneous one from the most probable noisy class can be constructed. Such an instruction can help any MLLMs with ICL ability to detect and rectify incorrect predictions of vision models. Through extensive experiments on ImageNet, WILDS, DomainBed, and other OOD datasets, we carefully validate the quantitative and qualitative effectiveness of our method. Our code is available at https://github.com/tmllab/Machine_Vision_Therapy.
MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance
KT Corporation, Republic of Korea（韩国KT公司）；Graduate School of Artificial Intelligence, POSTECH, Republic of Korea（POSTECH人工智能研究生院）；Computer Science and Engineering, POSTECH, Republic of Korea（POSTECH计算机科学与工程系）
In this paper, we propose a new multimodal image denoising approach to attenuate white Gaussian additive noise in a given image modality under the aid of a guidance image modality. The proposed coupled image denoising approach consists of two stages: coupled sparse coding and reconstruction. The first stage performs joint sparse transform for multimodal images with respect to a group of learned coupled dictionaries, followed by a shrinkage operation on the sparse representations. Then, in the second stage, the shrunken representations, together with coupled dictionaries, contribute to the reco…
A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy
Department of Radiation Oncology, Mayo Clinic, Phoenix, Arizona, USA（梅奥诊所放射肿瘤科）；School of Biomedical Engineering, ShanghaiTech University, Shanghai, China（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd., Shanghai 200230, China（上海联合影像智能科技有限公司）；Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China（上海人工智能实验室）；Shanghai Clinical Research and Trial Center, Shanghai 201210, China（上海临床研究与试验中心）
MID: A Self-supervised Multimodal Iterative Denoising Framework
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（自动化与智能感知学院，上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理重点实验室，中华人民共和国教育部）
Modeling Pedestrian Intrinsic Uncertainty for Multimodal Stochastic Trajectory Prediction via Energy Plan Denoising
School of Computing, Macquarie University（麦考瑞大学计算学院）；Data 61, CSIRO（联邦科学与工业研究组织Data 61研究院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）
Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification
The Guangdong Provincial Key Laboratory of Computational Intelligence and Cyberspace Information, the School of Computer Science and Engineering, South China University of Technology（广东省计算智能与网络信息重点实验室、计算机科学与工程学院、华南理工大学）；The Pazhou Laboratory（琶洲实验室）
Image synthesis under multi-modal priors is a useful and challenging task that has received increasing attention in recent years. A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities (i.e. priors) and corresponding outputs. In recent work, a variational auto-encoder (VAE) model was trained in a weakly supervised manner to address this challenge. Since the generative power of VAEs is usually limited, it is difficult for this method to synthesize images belonging to complex distributions. To this end, we propose a solution base…
LLM-driven Multimodal Target Volume Contouring in Radiation Oncology
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Yonsei University College of Medicine（延世大学医学院）；Yonsei University（延世大学）；Yongin Severance Hospital（龙仁世福兰斯医院）；Gangnam Severance Hospital（江南世福兰斯医院）；Oncosoft Inc.（Oncosoft公司）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））
A Unified Framework for Multimodal Image Reconstruction and Synthesis using Denoising Diffusion Models
Department of Computer Science and Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校计算机科学与工程系）；Mallinckrodt Institute of Radiology, Washington University in St. Louis（华盛顿大学圣路易斯分校马林克罗德特放射医学研究所）；Department of Electrical and Systems Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校电气与系统工程系）；Department of Neurology, Washington University in St. Louis（华盛顿大学圣路易斯分校神经病学系）；Department of Biomedical Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校生物医学工程系）；Division of Biology and Biomedical Sciences, Washington University in St. Louis（华盛顿大学圣路易斯分校生物学与生物医学科学 division）；Department of Electrical and Computer Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校电气与计算机工程系）
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence（佛罗伦萨大学媒体集成与传播中心（MICC））；University of Florence（佛罗伦萨大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of Pisa（比萨大学）
Optical coherence tomography (OCT) is a prevalent imaging technique for retina. However, it is affected by multiplicative speckle noise that can degrade the visibility of essential anatomical structures, including blood vessels and tissue layers. Although averaging repeated B-scan frames can significantly improve the signal-to-noise-ratio (SNR), this requires longer acquisition time, which can introduce motion artifacts and cause discomfort to patients. In this study, we propose a learning-based method that exploits information from the single-frame noisy B-scan and a pseudo-modality that is c…
Automatic segmentation of brain glioma from multimodal MRI scans plays a key role in clinical trials and practice. Unfortunately, manual segmentation is very challenging, time-consuming, costly, and often inaccurate despite human expertise due to the high variance and high uncertainty in the human annotations. In the present work, we develop an end-to-end deep-learning-based segmentation method using a multi-decoder architecture by jointly learning three separate sub-problems using a partly shared encoder. We also propose to apply smoothing methods to the input images to generate denoised vers…
Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Models
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
Out-of-Sight Embodied Agents: Multimodal Tracking, Sensor Fusion, and Trajectory Forecasting
Department of Electrical and Computer Engineering, Northeastern University（东北大学电气与计算机工程系）；Khoury College of Computer Sciences, Northeastern University（东北大学库里计算机科学学院）
Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Content
Mathematics and Statistics Department University of Texas at Dallas（德克萨斯大学达拉斯分校数学与统计学系）；Computer Science Department University of Texas at Dallas（德克萨斯大学达拉斯分校计算机科学系）；Naveen Jindal School of Management University of Texas at Dallas（德克萨斯大学达拉斯分校奈文·金达尔管理学院）
