Frontier models have demonstrated remarkable capabilities in understanding and reasoning with natural-language text, but they still exhibit major competency gaps in multimodal understanding and reasoning especially in high-value verticals such as biomedicine. Medical imaging report generation is a prominent example. Supervised fine-tuning can substantially improve performance, but they are prone to overfitting to superficial boilerplate patterns. In this paper, we introduce Universal Report Generation (UniRG) as a general framework for medical imaging report generation. By leveraging reinforcement learning as a unifying mechanism to directly optimize for evaluation metrics designed for end applications, UniRG can significantly improve upon supervised fine-tuning and attain durable generalization across diverse institutions and clinical practices. We trained UniRG-CXR on publicly available chest X-ray (CXR) data and conducted a thorough evaluation in CXR report generation with rigorous evaluation scenarios. On the authoritative ReXrank benchmark, UniRG-CXR sets new overall SOTA, outperforming prior state of the art by a wide margin. We release our model at https://huggingface.co/microsoft/UniRG-CXR.
SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis
College of Information and Intelligence, Hunan Agricultural University(湖南农业大学信息与智能学院);College of Artificial Intelligence, Nankai University(南开大学人工智能学院)
A review of deep learning-based information fusion techniques for multimodal medical image classification
Inserm(法国国家健康与医学研究院);University of Western Brittany(西布列塔尼大学);IMT Atlantique(大西洋电信学院);Sorbonne University(索邦大学);Lariboisière Hospital(拉里布瓦西耶尔医院);AP-HP(巴黎公共医疗集团);Paris Cité University(巴黎城市大学);CHRU Brest(布雷斯特大学中心医院);LaTIM UMR 1101(LaTIM联合研究单位1101)
Early Intervention for VFM-based Multimodal Medical Image Classification
Renmin University of China(中国人民大学);Beijing Key Laboratory for Intelligent Diagnosis of Fundus Diseases and Drug-Device R&D and Translation(眼底疾病智能诊断与药械研发转化北京市重点实验室)
本文提出EI框架,通过早期干预提升多模态影像疾病识别性能,引入MoR方法优化Vision Foundation Models适应,验证了在三个公开数据集上的有效性。
Seeking Necessary and Sufficient Information from Multimodal Medical Data
University College London, London, UK(伦敦大学学院);Imperial College London, London, UK(伦敦帝国学院);Mingdu Tech, China(明都科技);University of Bristol, Bristol, UK(布里斯托大学)
Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model
Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);College of AI, Tsinghua University(清华大学人工智能学院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)
UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis
Shanghai Artificial Intelligence Laboratory;Shanghai Innovation Institute;Shanghai Jiao Tong University;Shanghai Institute of Optics;Fudan University;University of Cambridge;Monash University;DAMO Academy, Alibaba Group;Imperial College London;The University of Hong Kong;The Hong Kong University of Science;Hupan Lab;The Chinese University of Hong Kong
Deep evidential fusion with uncertainty quantification and contextual discounting for multimodal medical image segmentation
Université de technologie de Compiègne(贡比涅技术大学);CNRS(法国国家科学研究中心);Université de Rouen Normandie(鲁昂诺曼底大学);Centre Henri Becquerel(亨利·贝克勒尔中心);Institut universitaire de France(法国大学研究院)
Unsupervised Multimodal 3D Medical Image Registration with Multilevel Correlation Balanced Optimization
College of Electrical and Information Engineering, Hunan University(湖南大学电气与信息工程学院);National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(湖南大学机器人视觉感知与控制技术国家工程研究中心);Cornell University(康奈尔大学)
Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis
Penn State University(宾夕法尼亚州立大学);Mississippi State University(密西西比州立大学);Northwest Missouri State University(西北密苏里州立大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Berkeley(加利福尼亚大学伯克利分校);Colorado State University(科罗拉多州立大学)
A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis
The University of Sydney(悉尼大学);Tencent AI Lab, Tencent(腾讯AI实验室);Guangzhou University of Chinese Medicine(广州中医药大学);The University of Wollongong(伍伦贡大学);The University of Adelaide(阿德莱德大学)
Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning
Department of Systems Engineering, Stevens Institute of Technology, Hoboken, New Jersey, USA(系统工程系,史蒂文斯理工学院);Department of Nuclear Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan(高雄长庚纪念医院核医学部);Institute of Statistics, National University of Kaohsiung, Kaohsiung, Taiwan(国立高雄大学统计研究所);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China(深圳先进技术研究院,中国科学院)
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute(伦斯勒理工学院);Wake Forest University School of Medicine(维克森林大学医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
Building RadiologyNET: Unsupervised annotation of a large-scale multimodal medical database
University of Rijeka – Faculty of Engineering(里耶卡大学工程学院);Center for Artificial Intelligence and Cybersecurity(人工智能与网络安全中心);Medical University of Graz(格拉茨医科大学)
Survey on deep learning in multimodal medical imaging for cancer detection
Massey University(梅西大学);Nantong University(南通大学);Zhejiang Engineering Research Center of Intelligent Medicine(浙江省智能医学工程研究中心);Jianpei Technology Co. Ltd(建培科技有限公司)
Deep reinforcement learning in medical imaging: A literature review
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Arkansas(阿肯色大学);University of Houston(休斯顿大学);INRIA, Sophia Antipolis-Mediterranean Centre(法国国家信息与自动化研究所索菲亚·安蒂波利斯地中海中心)
Multimodal Medical Image Binding via Shared Text Embeddings
The Hong Kong Polytechnic University(香港理工大学);Emory University(埃默里大学);The University of Hong Kong(香港大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);Johns Hopkins University(约翰霍普金斯大学)
ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
Hasso Plattner Institute for Digital Engineering, University of Potsdam(波茨坦大学哈索·普拉特纳数字工程研究所);TU Kaiserslautern(凯泽斯劳滕工业大学);Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院哈索·普拉特纳数字健康研究所)
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University(软件与微电子学院,北京大学);School of Computer Science, Peking University(计算机学院,北京大学);National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);Peking University Sixth Hospital(北京大学第六医院);Augusta University(奥古斯塔大学);Peking University First Hospital(北京大学第一医院)
ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue
PAII Inc.(PAII公司);Ping An Technology(平安科技);Ping An Healthcare and Technology Company Limited(平安医疗健康科技有限公司);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Tianjin University(天津大学)