Automatic radiology report generation is booming due to its huge application potential for the healthcare industry. However, existing computer vision and natural language processing approaches to tackle this problem are limited in two aspects. First, when extracting image features, most of them neglect multi-view reasoning in vision and model single-view structure of medical images, such as space-view or channel-view. However, clinicians rely on multi-view imaging information for comprehensive judgment in daily clinical diagnosis. Second, when generating reports, they overlook context reasoning with multi-modal information and focus on pure textual optimization utilizing retrieval-based methods. We aim to address these two issues by proposing a model that better simulates clinicians' perspectives and generates more accurate reports. Given the above limitation in feature extraction, we propose a Globally-intensive Attention (GIA) module in the medical image encoder to simulate and integrate multi-view vision perception. GIA aims to learn three types of vision perception: depth view, space view, and pixel view. On the other hand, to address the above problem in report generation, we explore how to involve multi-modal signals to generate precisely matched reports, i.e., how to integrate previously predicted words with region-aware visual content in next word prediction. Specifically, we design a Visual Knowledge-guided Decoder (VKGD), which can adaptively consider how much the model needs to rely on visual information and previously predicted text to assist next word prediction. Hence, our final Intensive Vision-guided Network (IVGN) framework includes a GIA-guided Visual Encoder and the VKGD. Experiments on two commonly-used datasets IU X-Ray and MIMIC-CXR demonstrate the superior ability of our method compared with other state-of-the-art approaches.
Trust It or Not: Confidence-Guided Automatic Radiology Report Generation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Duke University(杜克大学);The Chinese University of Hong Kong(香港中文大学);Brigham and Women’s Hospital, Harvard Medical School(布莱根妇女医院,哈佛医学院);Lenovo Research(联想研究院);Lenovo Ltd.(联想集团有限公司);University of North Carolina at Charlotte(北卡罗来纳大学夏洛特分校)
SGRNet: Spatially Guided Radiology Network for Structured Radiological Reporting of Head and Neck Cancer
University of Strasbourg(斯特拉斯堡大学);IHU Strasbourg(斯特拉斯堡大学医院研究所);Indian Institute of Technology (IIT) Madras(印度马德拉斯理工学院);All India Institute of Medical Sciences(全印度医学科学院)
Semantically Informed Salient Regions Guided Radiology Report Generation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(人工智能学院,北京邮电大学);Department of Radiology, Peking University Third Hospital(北京大学第三医院放射科)
Location-based Radiology Report-Guided Semi-supervised Learning for Prostate Cancer Detection
National Cancer Institute(美国国家癌症研究所);National Institutes of Health(美国国立卫生研究院);Clinical Center, National Institutes of Health(美国国立卫生研究院临床中心);Center for Cancer Research, National Cancer Institute(美国国家癌症研究所癌症研究中心)
A Self-Guided Framework for Radiology Report Generation
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen University General Hospital(深圳大学总医院)
Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
Memorial Sloan Kettering Cancer Center(纪念斯隆凯特琳癌症中心);Rochester Institute of Technology(罗切斯特理工学院);Institute of Research and Medical Care (IRCCS) SDN(研究与医疗护理研究所(IRCCS)SDN);University of Cagliari(卡利亚里大学)
Anatomy-aware Fine-grained Multimodal Fusion for Laryngopharyngeal Cancer T-Staging Prediction Using CT and Radiology Report
DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College(中国医学科学院北京协和医学院北京协和医院);Hupan Laboratory(湖畔实验室);Fuzhou University(福州大学);Eye & ENT Hospital, Fudan University(复旦大学附属眼耳鼻喉科医院);Zhongshan Hospital, Fudan University(复旦大学附属中山医院);Chang Gung Memorial Hospital(长庚纪念医院);Zhongshan Hospital(Xiamen), Fudan University(复旦大学附属中山医院厦门医院)
Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation
Imperial College London(帝国理工学院);The Ohio State University(俄亥俄州立大学);University of Hong Kong(香港大学);Hong Kong University of Science and Technology(香港科技大学)
R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generation
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);First Affiliated Hospital of Anhui University of Chinese Medicine(安徽中医药大学第一附属医院)
GLeVE: Graph-Guided Lesion Grounding with Proposal Verification in 3D CT
Zhejiang Key Laboratory of Space Information Sensing and Transmission(浙江空间信息感知与传输重点实验室);Hangzhou Dianzi University(杭州电子科技大学);Zhejiang University(浙江大学);Tsinghua University(清华大学);Children's Hospital, Zhejiang University School of Medicine(浙江大学医学院附属儿童医院)
Homogeneous Semantic Alignment and Hierarchical Expert Routing for Radiology Report Generation
Xinjiang University(新疆大学);Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center(新疆多模态智能处理与信息安全工程技术研究中心);Tsinghua University(清华大学)
CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generation
Institute for AI in Medicine (IKIM), University Hospital Essen (AöR)(人工智能医学研究所(IKIM)、埃森大学医院(AöR));Department of Nuclear Medicine, University Hospital Essen (AöR)(核医学系、埃森大学医院(AöR));Department of Physics, TU Dortmund(物理系、多特蒙德技术大学)
Instance-Guided Report Anchoring for Text-Free 3D Abnormality Segmentation in Chest CT
Siemens Medical Solutions USA, Inc.(西门子美国医疗解决方案公司);The Ohio State University(俄亥俄州立大学);University of California, Los Angeles(加州大学洛杉矶分校);Biomedical Imaging Research Institute, Cedars-Sinai Medical Center(西达赛奈医疗中心生物医学成像研究所);Yale University(耶鲁大学)
Interpretability-guided Data Augmentation for Robust Segmentation in Multi-centre Colonoscopy Data
The Netherlands Cancer Institute(荷兰癌症研究所);GROW School for Oncology and Developmental Biology, Maastricht University Medical Center(马斯特里赫特大学医学中心GROW肿瘤与发育生物学学院)
LLM-Guided Diagnostic Evidence Alignment for Medical Vision-Language Pretraining under Limited Pairing
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学阿利安斯商学院);Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系)
German Cancer Research Center (DKFZ)(德国癌症研究中心);Helmholtz Imaging, German Cancer Research Center (DKFZ)(亥姆霍兹影像,德国癌症研究中心);Helmholtz Association(亥姆霍兹联合会);Heidelberg University(海德堡大学)
EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledge
Xidian University(西安电子科技大学);Shanghai Jiao Tong University School of Medicine(上海交通大学医学院);Ruijin Hospital(瑞金医院);Shanghai Institute of Traumatology and Orthopedics(上海市伤骨科研究所);Harbin Medical University(哈尔滨医科大学);The First Affiliated Hospital of Harbin Medical University(哈尔滨医科大学附属第一医院);Brown University(布朗大学)
Representation-guided in-context learning for medical image interpretation with multimodal large language models
Schepens Eye Research Institute of Massachusetts Eye and Ear(麻省眼耳医院谢本斯眼科研究所);Harvard Medical School(哈佛医学院);Massachusetts Eye and Ear(麻省眼耳医院);National Taiwan University Hospital(台湾大学医院);Massachusetts Institute of Technology(麻省理工学院);Northwestern University(西北大学);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学肯普纳自然与人工智能研究所);Harvard University(哈佛大学);Harvard Data Science Initiative, Harvard University(哈佛大学数据科学倡议)
Mask-guided Data Augmentation for Multiparametric MRI Generation with a Rare Hepatocellular Carcinoma
Universidad Industrial de Santander(桑坦德工业大学);University of Toulouse(图卢兹大学);Université Paul Sabatier Toulouse 3(图卢兹第三大学(保罗·萨巴蒂埃大学));Bicêtre University Hospital(比塞特尔大学医院);Assistance publique des hôpitaux de Paris(巴黎公共援助医院);Université Claude Bernard Lyon 1(里昂第一大学(克洛德·贝尔纳里昂第一大学))
Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings
The Center for Smart Health, School of Nursing, the Hong Kong Polytechnic University, Hong Kong, China(香港理工大学智能健康中心,护理学院,中国香港);Research Institute for Smart Ageing, the Hong Kong Polytechnic University, Hong Kong, China(香港理工大学智能老龄化研究 institute,中国香港);School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China(清华大学生物医学工程学院,清华大学,北京,中国);Queen Mary Hospital, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China(香港大学李嘉诚医学院Queen Mary医院,中国香港)
MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images
Boston University(波士顿大学);Stanford University(斯坦福大学);Carnegie Mellon University(卡内基梅隆大学);University of Pittsburgh Medical Center(匹兹堡大学医学中心);University of Pittsburgh(匹兹堡大学)
Hardware-Efficient Guided Image Filtering For Multi-Label Problem
Nanjing University of Science and Technology(南京理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)