In this study, we propose a new activation function, called Adaptive Smooth Activation Unit (ASAU), tailored for optimized gradient propagation, thereby enhancing the proficiency of convolutional networks in medical image analysis. We apply this new activation function to two important and commonly used general tasks in medical image analysis: automatic disease diagnosis and organ segmentation in CT and MRI. Our rigorous evaluation on the RadImageNet abdominal/pelvis (CT and MRI) dataset and Liver Tumor Segmentation Benchmark (LiTS) 2017 demonstrates that our ASAU-integrated frameworks not only achieve a substantial (4.80\%) improvement over ReLU in classification accuracy (disease detection) on abdominal CT and MRI but also achieves 1\%-3\% improvement in dice coefficient compared to widely used activations for `healthy liver tissue' segmentation. These improvements offer new baselines for developing a diagnostic tool, particularly for complex, challenging pathologies. The superior performance and adaptability of ASAU highlight its potential for integration into a wide range of image classification and segmentation tasks.
School of Biomedical Engineering, King’s College London（伦敦国王学院生物医学工程学院）；King’s College Hospital（国王学院医院）；Institute of Psychiatry, Psychology & Neuroscience, King’s College London（伦敦国王学院精神病学、心理学与神经科学研究所）；Centre for Medical Image Computing, Dementia Research, University College London（伦敦大学学院医学影像计算中心（痴呆研究））
Labelling large datasets for training high-capacity neural networks is a major obstacle to the development of deep learning-based medical imaging applications. Here we present a transformer-based network for magnetic resonance imaging (MRI) radiology report classification which automates this task by assigning image labels on the basis of free-text expert radiology reports. Our model's performance is comparable to that of an expert radiologist, and better than that of an expert physician, demonstrating the feasibility of this approach. We make code available online for researchers to label the…
Emory University（埃默里大学）；Indiana University Purdue University（印第安纳大学与普渡大学）；Mentor Graphics India Pvt. Ltd.（明导国际印度私人有限公司）；CISSS Lanaudiere（拉纳迪耶综合健康与社会服务中心）；Laval University（拉瓦尔大学）
Executing machine learning (ML) pipelines in real-time on radiology images is hard due to the limited computing resources in clinical environments and the lack of efficient data transfer capabilities to run them on research clusters. We propose Niffler, an integrated framework that enables the execution of ML pipelines at research clusters by efficiently querying and retrieving radiology images from the Picture Archiving and Communication Systems (PACS) of the hospitals. Niffler uses the Digital Imaging and Communications in Medicine (DICOM) protocol to fetch and store imaging data and provide…
National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）
Despite tremendous progress in computer vision, there has not been an attempt for machine learning on very large-scale medical image databases. We present an interleaved text/image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's Picture Archiving and Communication System. With natural language processing, we mine a collection of representative ~216K two-dimensional key images selected by clinicians for diagnostic reference, and match the images with their descriptions in an automated manner. Our system inter…
Anatomy-aware Fine-grained Multimodal Fusion for Laryngopharyngeal Cancer T-Staging Prediction Using CT and Radiology Report
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院北京协和医院）；Hupan Laboratory（湖畔实验室）；Fuzhou University（福州大学）；Eye & ENT Hospital, Fudan University（复旦大学附属眼耳鼻喉科医院）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）；Chang Gung Memorial Hospital（长庚纪念医院）；Zhongshan Hospital(Xiamen), Fudan University（复旦大学附属中山医院厦门医院）
National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）；NVIDIA（英伟达）
Radiologists in their daily work routinely find and annotate significant abnormalities on a large number of radiology images. Such abnormalities, or lesions, have collected over years and stored in hospitals' picture archiving and communication systems. However, they are basically unsorted and lack semantic annotations like type and location. In this paper, we aim to organize and explore them by learning a deep feature representation for each lesion. A large-scale and comprehensive dataset, DeepLesion, is introduced for this task. DeepLesion contains bounding boxes and size measurements of ove…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Rochester Institute of Technology（罗切斯特理工学院）；Institute of Research and Medical Care (IRCCS) SDN（研究与医疗护理研究所（IRCCS）SDN）；University of Cagliari（卡利亚里大学）
Artificial intelligence (AI) has been successful at solving numerous problems in machine perception. In radiology, AI systems are rapidly evolving and show progress in guiding treatment decisions, diagnosing, localizing disease on medical images, and improving radiologists' efficiency. A critical component to deploying AI in radiology is to gain confidence in a developed system's efficacy and safety. The current gold standard approach is to conduct an analytical validation of performance on a generalization dataset from one or more institutions, followed by a clinical validation study of the s…
Faculty of Science, Technology and Communication（科学、技术与传播学院）
Within this thesis we propose a platform for combining Augmented Reality (AR) hardware with machine learning in a user-oriented pipeline, offering to the medical staff an intuitive 3D visualization of volumetric Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) medical image segmentations inside the AR headset, that does not need human intervention for loading, processing and segmentation of medical images. The AR visualization, based on Microsoft HoloLens, employs a modular and thus scalable frontend-backend architecture for real-time visualizations on multiple AR headsets. As Con…
Duke University（杜克大学）；Department of Radiology, Duke University（杜克大学放射科）；Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）；Duke Medical Physics Program, Duke University（杜克大学医学物理项目）；Center for Advanced Magnetic Resonance Development, Duke University（杜克大学先进磁共振发展中心）
Deep learning is a branch of artificial intelligence where networks of simple interconnected units are used to extract patterns from data in order to solve complex problems. Deep learning algorithms have shown groundbreaking performance in a variety of sophisticated tasks, especially those related to images. They have often matched or exceeded human performance. Since the medical field of radiology mostly relies on extracting useful information from images, it is a very natural application area for deep learning, and research in this area has rapidly grown in recent years. In this article, we …
Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation
Imperial College London（帝国理工学院）；The Ohio State University（俄亥俄州立大学）；University of Hong Kong（香港大学）；Hong Kong University of Science and Technology（香港科技大学）
Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医科大学）；Student Research Committee（学生研究委员会）；School of Allied Medical Science（Allied Medical Science学院）；Department of Health Information Technology and Management（卫生信息技术与管理系）
Purpose: Early detection and diagnosis of Covid-19 and accurate separation of patients with non-Covid-19 cases at the lowest cost and in the early stages of the disease are one of the main challenges in the epidemic of Covid-19. Concerning the novelty of the disease, the diagnostic methods based on radiological images suffer shortcomings despite their many uses in diagnostic centers. Accordingly, medical and computer researchers tended to use machine-learning models to analyze radiology images. Methods: Present systematic review was conducted by searching three databases of PubMed, Scopus, and…
Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset
Stanford University（斯坦福大学）；Stanford Center for Artificial Intelligence in Medicine and Imaging（斯坦福大学医学与成像人工智能中心）；Department of Electrical Engineering（电气工程系）；University of California Berkeley（加州大学伯克利分校）；Department of Radiology（放射科）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Hospital Israelita Albert Einstein（以色列特医院阿尔伯特·爱因斯坦医院）
National Institutes of Health (NIH) Clinical Center（美国国立卫生研究院临床中心）；National Center for Biotechnology Information, National Library of Medicine, NIH（美国国立卫生研究院国家医学图书馆国家生物技术信息中心）
In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and write sentences in the radiology report to describe them. In this paper, we study the lesion description or annotation problem as an important step of computer-aided diagnosis (CAD). Given a lesion image, our aim is to predict multiple relevant labels, such as the lesion's body part, type, and attributes. To address this problem, we define a set of 145 labels based on RadLex to describe a large variety of lesions in the DeepLesion dataset. We directly mine training labels fr…
John Hopkins University（约翰霍普金斯大学）；University of Bologna（博洛尼亚大学）；Italian Institute of Technology（意大利技术研究院）；University of California, San Francisco（旧金山大学）；University of California, Berkeley（伯克利大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology
Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）；Clinic of Radiology and Nuclear Medicine, University Hospital Basel（巴塞尔大学医院放射科与核医学科）；Department of Oncology, Johns Hopkins School of Medicine（约翰霍普金斯医学院肿瘤科）
AI-Driven Radiology Report Generation for Traumatic Brain Injuries
Higher School of Digital Economics, Manouba, University of Manouba, Tunisia（数字经济学高等学院，曼布巴，曼布巴大学，突尼斯）；Higher Institute of Management of Tunis, University of Tunis, Tunisia（突尼斯管理高等学院，突尼斯大学，突尼斯）；Emergency Department Charles Nicolle Hospital, Tunis El Manar University, Tunisia（查尔斯·尼科勒医院急诊部，突尼斯埃尔曼纳大学，突尼斯）
Structure is Supervision: Multiview Masked Autoencoders for Radiology
Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Department of Computer Science, UC Irvine（加州大学尔湾分校计算机科学系）；Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）
University of Rochester（罗切斯特大学）；Futurewei Technologies, Inc.（未来 Wei 技术公司）
Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limite…
Radiology reports are an important means of communication between radiologists and other physicians. These reports express a radiologist's interpretation of a medical imaging examination and are critical in establishing a diagnosis and formulating a treatment plan. In this paper, we propose a Bi-directional convolutional neural network (Bi-CNN) model for the interpretation and classification of mammograms based on breast density and chest radiographic radiology reports based on the basis of chest pathology. The proposed approach helps to organize databases of radiology reports, retrieve them e…
ReportMedSAM: Guiding Segmentation Through Radiology Reports
School of Engineering, College of Engineering and Physical Sciences, University of Birmingham（工程学院，工程与物理科学学院，伯明翰大学）；Department of Cancer and Genomic Sciences, College of Medicine and Health, University of Birmingham（癌症与基因组科学系，医学与健康学院，伯明翰大学）；Department of Computer Science, Faculty of Science and Engineering, University of Manchester（计算机科学系，科学与工程学院，曼彻斯特大学）；The Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine, and Health, University of Manchester（信息学、成像与数据科学部，生物、医学与健康学院，曼彻斯特大学）
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation
Sun Yat-Sen Memorial Hospital（孙逸仙纪念医院）；Sun Yat-Sen University（中山大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）；Xunfei Healthcare Technology Co., Ltd.（讯飞医疗科技股份有限公司）
The University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；Shandong First Medical University（山东第一医科大学）；Shandong Academy of Medical Sciences（山东省医学科学院）；Shandong Cancer Hospital and Institute（山东省肿瘤医院）
There is an urgent need for automated methods to assist accurate and effective assessment of COVID-19. Radiology and nucleic acid test (NAT) are complementary COVID-19 diagnosis methods. In this paper, we present an end-to-end multitask learning (MTL) framework (COVID-MTL) that is capable of automated and simultaneous detection (against both radiology and NAT) and severity assessment of COVID-19. COVID-MTL learns different COVID-19 tasks in parallel through our novel random-weighted loss function, which assigns learning weights under Dirichlet distribution to prevent task dominance; our new 3D…
Renal cell carcinoma (RCC) is a common cancer that varies in clinical behavior. Indolent RCC is often low-grade without necrosis and can be monitored without treatment. Aggressive RCC is often high-grade and can cause metastasis and death if not promptly detected and treated. While most kidney cancers are detected on CT scans, grading is based on histology from invasive biopsy or surgery. Determining aggressiveness on CT images is clinically important as it facilitates risk stratification and treatment planning. This study aims to use machine learning methods to identify radiology features tha…
Philips Innovation Labs Rus（飞利浦俄罗斯创新实验室）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Sber AI Lab（俄联邦储蓄银行人工智能实验室）
The automatic clinical caption generation problem is referred to as proposed model combining the analysis of frontal chest X-Ray scans with structured patient information from the radiology records. We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed combination of these models generates a textual summary with the essential information about pathologies found, their location, and the 2D heatmaps localizing each pathology on the original X-Ray scans. The proposed model is tested on two medical datasets, th…
National Center for Biotechnology Information, National Library of Medicine（美国国家医学图书馆国家生物技术信息中心）
In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and describe them in the radiology report. In this paper, we study the lesion description or annotation problem. Given a lesion image, our aim is to predict a comprehensive set of relevant labels, such as the lesion's body part, type, and attributes, which may assist downstream fine-grained diagnosis. To address this task, we first design a deep learning module to extract relevant semantic labels from the radiology reports associated with the lesion images. With the images and t…
PETWB-REP: A Multi-Cancer Whole-Body FDG PET/CT and Radiology Report Dataset for Medical Imaging Research
PET Center, Huashan Hospital, Fudan University（复旦大学附属华山医院PET中心）；Shanghai Academy of Artificial Intelligence for Science（上海科学智能研究院）；Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）；Shanghai Universal Medical Imaging Diagnostic Center（上海环球医学影像诊断中心）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）
Understanding of spatial attributes is central to effective 3D radiology image analysis where crop-based learning is the de facto standard. Given an image patch, its core spatial properties (e.g., position & orientation) provide helpful priors on expected object sizes, appearances, and structures through inherent anatomical consistencies. Spatial correspondences, in particular, can effectively gauge semantic similarities between inter-image regions, while their approximate extraction requires no annotations or overbearing computational costs. However, recent 3D contrastive learning approaches …
Curia: A Multi-Modal Foundation Model for Radiology
Raidium；Hôpital Européen Georges Pompidou, AP-HP（欧洲乔治·蓬皮杜医院, AP-HP）；Université Paris-Cité（巴黎城市大学）；INRIA（法国国家信息与自动化技术研究所）；INSERM（法国国家卫生与医学研究中心）；Beaujon Hospital（贝琼医院）；Medical University of South Carolina（南卡罗来纳医科大学）；Columbia University Irving Medical Center（哥伦比亚大学伊万斯医学中心）；Centre Cardiologique du Nord（北心脑血管中心）
National Institute of Technology Karnataka Surathkal（卡纳塔克邦国家技术学院苏拉特卡尔分校）
In clinical diagnosis, diagnostic images that are obtained from the scanning devices serve as preliminary evidence for further investigation in the process of delivering quality healthcare. However, often the medical image may contain fault artifacts, introduced due to noise, blur and faulty equipment. The reason for this may be the low-quality or older scanning devices, the test environment or technicians lack of training etc; however, the net result is that the process of fast and reliable diagnosis is hampered. Resolving these issues automatically can have a significant positive impact in a…
Christian Brothers University（基督教兄弟大学）；University of Dayton（代顿大学）
With a Coronavirus disease (COVID-19) case count exceeding 10 million worldwide, there is an increased need for a diagnostic capability. The main variables in increasing diagnostic capability are reduced cost, turnaround or diagnosis time, and upfront equipment cost and accessibility. Two candidates for machine learning COVID-19 diagnosis are Computed Tomography (CT) scans and plain chest X-rays. While CT scans score higher in sensitivity, they have a higher cost, maintenance requirement, and turnaround time as compared to plain chest X-rays. The use of portable chest X-radiograph (CXR) is rec…
LEAVS: An LLM-based Labeler for Abdominal CT Supervision
National Institutes of Health Clinical Center（美国国家卫生研究院临床中心）；Icahn School of Medicine at Mount Sinai（伊坎医学院）；Walter Reed National Military Medical Center（沃尔特·里德国家军事医疗中心）
University of Stuttgart（斯图加特大学）；University of Tübingen（图宾根大学）；King’s College London（伦敦国王学院）
Individuals age differently depending on a multitude of different factors such as lifestyle, medical history and genetics. Often, the global chronological age is not indicative of the true ageing process. An organ-based age estimation would yield a more accurate health state assessment. In this work, we propose a new deep learning architecture for organ-based age estimation based on magnetic resonance images (MRI). The proposed network is a 3D convolutional neural network (CNN) with increased depth and width made possible by the hybrid utilization of inception and fire modules. We apply the pr…
SGRNet: Spatially Guided Radiology Network for Structured Radiological Reporting of Head and Neck Cancer
University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Indian Institute of Technology (IIT) Madras（印度马德拉斯理工学院）；All India Institute of Medical Sciences（全印度医学科学院）
A Synthetic Data-Driven Radiology Foundation Model for Pan-tumor Clinical Diagnosis
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Lab（上海人工智能实验室）；The First Hospital of China Medical University（中国医科大学附属第一医院）；Harvard Medical School（哈佛医学院）
Mass General Brigham（麻省总医院布里格姆医疗系统）；Harvard Medical School（哈佛医学院）；Maastricht University（马斯特里赫特大学）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Technical University of Munich（慕尼黑工业大学）；German Heart Center（德国心脏中心）
Automated abdominal multi-organ segmentation is a crucial yet challenging task in the computer-aided diagnosis of abdominal organ-related diseases. Although numerous deep learning models have achieved remarkable success in many medical image segmentation tasks, accurate segmentation of abdominal organs remains challenging, due to the varying sizes of abdominal organs and the ambiguous boundaries among them. In this paper, we propose a boundary-aware network (BA-Net) to segment abdominal organs on CT scans and MRI scans. This model contains a shared encoder, a boundary decoder, and a segmentati…
