Existing promptable segmentation methods in the medical imaging field primarily consider either textual or visual prompts to segment relevant objects, yet they often fall short when addressing anomalies in medical images, like tumors, which may vary greatly in shape, size, and appearance. Recognizing the complexity of medical scenarios and the limitations of textual or visual prompts, we propose a novel dual-prompt schema that leverages the complementary strengths of visual and textual prompts for segmenting various organs and tumors. Specifically, we introduce CAT, an innovative model that Coordinates Anatomical prompts derived from 3D cropped images with Textual prompts enriched by medical domain knowledge. The model architecture adopts a general query-based design, where prompt queries facilitate segmentation queries for mask prediction. To synergize two types of prompts within a unified framework, we implement a ShareRefiner, which refines both segmentation and prompt queries while disentangling the two types of prompts. Trained on a consortium of 10 public CT datasets, CAT demonstrates superior performance in multiple segmentation tasks. Further validation on a specialized in-house dataset reveals the remarkable capacity of segmenting tumors across multiple cancer stages. This approach confirms that coordinating multimodal prompts is a promising avenue for addressing complex scenarios in the medical domain.
The University of Hong Kong（香港大学）；Tencent（腾讯）；Huazhong University of Science and Technology（华中科技大学）
Learning by imitation is one of the most significant abilities of human beings and plays a vital role in human's computational neural system. In medical image analysis, given several exemplars (anchors), experienced radiologist has the ability to delineate unfamiliar organs by imitating the reasoning process learned from existing types of organs. Inspired by this observation, we propose OrganNet which learns a generalized organ concept from a set of annotated organ classes and then transfer this concept to unseen classes. In this paper, we show that such process can be integrated into the one-…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Artificial Intelligence Research Institute (AIRI)（人工智能研究所（AIRI））；Institute for Information Transmission Problems（信息传输问题研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）
PAII Inc.（PAII公司）；The First Affiliated Hospital Zhejiang University（浙江大学附属第一医院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Ping An Insurance Company of China（中国平安保险（集团）股份有限公司）；Zhejiang Provincial People’s Hospital（浙江省人民医院）
Lymph node station (LNS) delineation from computed tomography (CT) scans is an indispensable step in radiation oncology workflow. High inter-user variabilities across oncologists and prohibitive laboring costs motivated the automated approach. Previous works exploit anatomical priors to infer LNS based on predefined ad-hoc margins. However, without voxel-level supervision, the performance is severely limited. LNS is highly context-dependent - LNS boundaries are constrained by anatomical organs - we formulate it as a deep spatial and contextual parsing problem via encoded anatomical organs. Thi…
Imperial College London（帝国理工学院）；Universitat Pompeu Fabra（庞培法布拉大学）；Galgo Medical S.L.（Galgo医疗有限公司）；Nara Institute of Science and Technology (NAIST)（奈良先端科学技术大学院大学（NAIST））；Institució Catalana de Recerca i Estudis Avancats (ICREA)（加泰罗尼亚研究与高级研究机构（ICREA））；Children’s National Health System（儿童国家卫生系统）；George Washington University（乔治华盛顿大学）
The medical image analysis field has traditionally been focused on the development of organ-, and disease-specific methods. Recently, the interest in the development of more 20 comprehensive computational anatomical models has grown, leading to the creation of multi-organ models. Multi-organ approaches, unlike traditional organ-specific strategies, incorporate inter-organ relations into the model, thus leading to a more accurate representation of the complex human anatomy. Inter-organ relations are not only spatial, but also functional and physiological. Over the years, the strategies 25 propo…
Université Côte d’Azur（蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；Centre Antoine Lacassagne（安托万·拉卡桑中心）
Planning of radiotherapy involves accurate segmentation of a large number of organs at risk, i.e. organs for which irradiation doses should be minimized to avoid important side effects of the therapy. We propose a deep learning method for segmentation of organs at risk inside the brain region, from Magnetic Resonance (MR) images. Our system performs segmentation of eight structures: eye, lens, optic nerve, optic chiasm, pituitary gland, hippocampus, brainstem and brain. We propose an efficient algorithm to train neural networks for an end-to-end segmentation of multiple and non-exclusive class…
AbdomenGen: Sequential Volume-Conditioned Diffusion Framework for Abdominal Anatomy Generation
Center for Virtual Imaging Trials, Dept. of Radiology, Duke University School of Medicine（虚拟成像试验中心，放射科，杜克大学医学部）；Dept. of Electrical and Computer Engineering, Pratt School of Engineering, Duke University（电气与计算机工程系，普拉特工程学院，杜克大学）；Dept. of Physics, Trinity School of Arts and Sciences, Duke University（物理系，特里尼蒂艺术与科学学院，杜克大学）
AbdomenGen提出了一种序列体积条件扩散框架，通过Volume Control Scalar实现可控的腹部解剖生成，实现了高几何保真度和多器官解耦调节。
AG-CRC: Anatomy-Guided Colorectal Cancer Segmentation in CT with Imperfect Anatomical Knowledge
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Digital Medicine Innovation Center（上海数字医学创新中心）；Ruijin Hospital（瑞金医院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
Anatomy-aware Fine-grained Multimodal Fusion for Laryngopharyngeal Cancer T-Staging Prediction Using CT and Radiology Report
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院北京协和医院）；Hupan Laboratory（湖畔实验室）；Fuzhou University（福州大学）；Eye & ENT Hospital, Fudan University（复旦大学附属眼耳鼻喉科医院）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）；Chang Gung Memorial Hospital（长庚纪念医院）；Zhongshan Hospital(Xiamen), Fudan University（复旦大学附属中山医院厦门医院）
CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generation
Institute for AI in Medicine (IKIM), University Hospital Essen (AöR)（人工智能医学研究所（IKIM）、埃森大学医院（AöR））；Department of Nuclear Medicine, University Hospital Essen (AöR)（核医学系、埃森大学医院（AöR））；Department of Physics, TU Dortmund（物理系、多特蒙德技术大学）
ORACLE-CT: Anatomy-Aware Support Pooling for CT Classification
Center for Virtual Imaging Trials, RAI Labs, Department of Radiology, Duke University（虚拟成像试验中心，RAI实验室，放射学系，杜克大学）；Electrical and Computer Engineering, Pratt School of Engineering, Duke University（电气与计算机工程，工程学院，杜克大学）；Department of Mathematics, Trinity College of Arts & Sciences, Duke University（数学系，艺术与科学学院，杜克大学）；Department of Radiology and Imaging Sciences, University of Arizona College of Medicine（放射学与影像科学系，亚利桑那大学医学院）
SAME++: A Self-supervised Anatomical eMbeddings Enhanced medical image registration framework using stable sampling and regularized transformation
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Alibaba Group（阿里巴巴集团）；Johns Hopkins University（约翰斯·霍普金斯大学）；The First Affiliated Hospital of Zhejiang University（浙江大学附属第一医院）
Devil is in Details: Locality-Aware 3D Abdominal CT Volume Generation for Self-Supervised Organ Segmentation
National Engineering Research Center for Multimedia Software（多媒体软件国家工程研究中心）；Hubei Key Laboratory of Multimedia and Network Communication Engineering（湖北省多媒体与网络通信工程重点实验室）；Institute of Artificial Intelligence（人工智能研究院）；School of Computer Science（计算机科学学院）
Accurate and robust segmentation of small organs in whole-body MRI is difficult due to anatomical variation and class imbalance. Recent deep network based approaches have demonstrated promising performance on abdominal multi-organ segmentations. However, the performance on small organs is still suboptimal as these occupy only small regions of the whole-body volumes with unclear boundaries and variable shapes. A coarse-to-fine, hierarchical strategy is a common approach to alleviate this problem, however, this might miss useful contextual information. We propose a two-stage approach with weight…
seg2med: a bridge from artificial anatomy to multimodal medical images
Computer Assisted Clinical Medicine, Medical Faculty Mannheim, Heidelberg University（计算机辅助临床医学，曼海姆医学院，海德堡大学）；Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nuremberg（模式识别实验室，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Department of Radiology and Nuclear Medicine, University Medical Center Mannheim（放射学与核医学系，曼海姆大学医学中心）；Mannheim Institute for Intelligent Systems in Medicine, Medical Faculty Mannheim, Heidelberg University（曼海姆智能医学研究所，曼海姆医学院，海德堡大学）；Optical Bioimaging Laboratory, Department of Biomedical Engineering, College of Design and Engineering, National University of Singapore（光学生物成像实验室，生物医学工程系，设计与工程学院，新加坡国立大学）
University of Waterloo（滑铁卢大学）；Suez Canal University（苏伊士运河大学）
Quantifying the accuracy of segmentation and manual delineation of organs, tissue types and tumors in medical images is a necessary measurement that suffers from multiple problems. One major shortcoming of all accuracy measures is that they neglect the anatomical significance or relevance of different zones within a given segment. Hence, existing accuracy metrics measure the overlap of a given segment with a ground-truth without any anatomical discrimination inside the segment. For instance, if we understand the rectal wall or urethral sphincter as anatomical zones, then current accuracy measu…
AnyCXR: Human Anatomy Segmentation of Chest X-ray at Any Acquisition Position using Multi-stage Domain Randomized Synthetic Data with Imperfect Annotations and Conditional Joint Annotation Regularization Learning
Department of Radiology, Northwestern University, Chicago, IL, USA（西北大学放射科）；Data Science Institute, Vanderbilt University, Nashville, TN, USA（范德比尔特大学数据科学研究院）；Department of Orthopedics, The Second Hospital of Shanxi Medical University, Taiyuan, Shanxi, P.R. China（山西医科大学第二医院骨科）；Second Clinical Medical College, Shanxi Medical University, Taiyuan, Shanxi, P.R. China（山西医科大学第二临床医学院）；School of Computer Science, University of Sydney, Sydney, NSW, Australia（悉尼大学计算机科学学院）
Accurate automatic organ segmentation is an important yet challenging problem for medical image analysis. The pancreas is an abdominal organ with very high anatomical variability. This inhibits traditional segmentation methods from achieving high accuracies, especially compared to other organs such as the liver, heart or kidneys. In this paper, we present a holistic learning approach that integrates semantic mid-level cues of deeply-learned organ interior and boundary maps via robust spatial aggregation using random forest. Our method generates boundary preserving pixel-wise class labels for p…
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Georgia Institute of Technology（佐治亚理工学院）
Thoracic disease detection from chest radiographs using deep learning methods has been an active area of research in the last decade. Most previous methods attempt to focus on the diseased organs of the image by identifying spatial regions responsible for significant contributions to the model's prediction. In contrast, expert radiologists first locate the prominent anatomical structures before determining if those regions are anomalous. Therefore, integrating anatomical knowledge within deep learning models could bring substantial improvement in automatic disease classification. Motivated by …
PromptForSegCXR: Prompt-Driven Multi-Organ and Multi-Disease Segmentation in Chest X-rays using a Multi-stage Fusion Mechanism
1 ,5Department of Computer Science \& Engineering,\ University of Engineering \& Technology, Rajshahi-6204, Bangladesh；2 Department of Electrical；Electronic Engineering,\ University of Engineering；Bachelor of Surgery,\ Suhrawardy Medical College , Sher-E-Bangla Nagar, Dhaka-1207；6 Department of Biomedical Engineering,\ University of Engineering
Improving Clinical Target Volume Segmentation Accuracy using Anatomical Priors and Active Learning for the AGITG TOPGEAR Clinical Trial
University of New South Wales（新南威尔士大学）；Ingham Institute for Applied Medical Research（英厄姆应用医学研究所）；Liverpool and Macarthur Cancer Therapy Centres（利物浦与麦克阿瑟癌症治疗中心）；Peter MacCallum Cancer Centre（彼得·麦卡勒姆癌症中心）；University of Melbourne（墨尔本大学）；CSIRO Australian e-Health Research Centre（联邦科学与工业研究组织澳大利亚电子健康研究中心）；University of Sydney（悉尼大学）；The University of Western Australia（西澳大学）；Sir Charles Gardiner Hospital（查尔斯·加德纳爵士医院）；University of Wisconsin（威斯康星大学）；Radformation Inc.（Radformation公司）
School of Psychology, The University of Queensland（昆士兰大学心理学院）；Queensland Brain Institute, The University of Queensland（昆士兰大学昆士兰脑研究所）；Centre for Advanced Imaging, The University of Queensland（昆士兰大学高级成像中心）
Whether it be in a man-made machine or a biological system, form and function are often directly related. In the latter, however, this particular relationship is often unclear due to the intricate nature of biology. Here we developed a geometric deep learning model capable of exploiting the actual structure of the cortex to learn the complex relationship between brain function and anatomy from structural and functional MRI data. Our model was not only able to predict the functional organization of human visual cortex from anatomical properties alone, but it was also able to predict nuanced var…
Gordon Center for Medical Imaging, Massachusetts General Hospital and Harvard Medical School（马萨诸塞州总医院与哈佛医学院戈登医学影像中心）
Lesions or organ boundaries visible through medical imaging data are often ambiguous, thus resulting in significant variations in multi-reader delineations, i.e., the source of aleatoric uncertainty. In particular, quantifying the inter-observer variability of manual annotations with Magnetic Resonance (MR) Imaging data plays a crucial role in establishing a reference standard for various diagnosis and treatment tasks. Most segmentation methods, however, simply model a mapping from an image to its single segmentation map and do not take the disagreement of annotators into consideration. In ord…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
Performing coarse-to-fine abdominal multi-organ segmentation facilitates to extract high-resolution segmentation minimizing the lost of spatial contextual information. However, current coarse-to-refine approaches require a significant number of models to perform single organ refine segmentation corresponding to the extracted organ region of interest (ROI). We propose a coarse-to-fine pipeline, which starts from the extraction of the global prior context of multiple organs from 3D volumes using a low-resolution coarse network, followed by a fine phase that uses a single refined model to segment…
Shanghai Jiao Tong University（上海交通大学）；University of Sydney（悉尼大学）；Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院瑞金医院）
Motion estimation is a fundamental step in dynamic medical image processing for the assessment of target organ anatomy and function. However, existing image-based motion estimation methods, which optimize the motion field by evaluating the local image similarity, are prone to produce implausible estimation, especially in the presence of large motion. In this study, we provide a novel motion estimation framework of Dense-Sparse-Dense (DSD), which comprises two stages. In the first stage, we process the raw dense image to extract sparse landmarks to represent the target organ anatomical topology…
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
Ultrasound imaging is generally employed for real-time investigation of internal anatomy of the human body for disease identification. Delineation of the anatomical boundary of organs and pathological lesions is quite challenging due to the stochastic nature of speckle intensity in the images, which also introduces visual fatigue for the observer. This paper introduces a fully convolutional neural network based method to segment organ and pathologies in ultrasound volume by learning the spatial-relationship between closely related classes in the presence of stochastically varying speckle inten…
Two-View Topogram-Based Anatomy-Guided CT Reconstruction for Prospective Risk Minimization
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡大学模式识别实验室）；Division of X-Ray Imaging and Computed Tomography, German Cancer Research Center (DKFZ)（德国癌症研究中心X射线成像与计算机断层扫描部门）；Department of Physics and Astronomy, Ruprecht-Karls-University Heidelberg（海德堡大学物理与天文学系）；Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡大学埃尔朗根大学医院放射肿瘤科）；Department of Radiology and Nuclear Medicine, Klinikum Nürnberg, Paracelsus Medical University（帕拉塞尔苏斯医科大学纽伦堡医院放射与核医学科）
CS-MUNet: A Channel-Spatial Dual-Stream Mamba Network for Multi-Organ Segmentation
School of Software Engineering（软件工程学院）；The Chinese University of Hong Kong（香港中文大学）；Yunnan Provincial Key Laboratory of Software Engineering（云南省软件工程重点实验室）
Anatomy-Informed Deep Learning for Abdominal Aortic Aneurysm Segmentation
Faculty of Mathematics, Informatics and Technology, University of Applied Sciences Koblenz（科布伦茨应用科学大学数学、信息学与技术学院）；Department of Radiology, Herz-Jesu-Krankenhaus（赫茨-耶苏医院放射科）
The University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）；Cedars Sinai Medical Center（西达赛奈医疗中心）
Magnetic resonance (MR) protocols rely on several sequences to assess pathology and organ status properly. Despite advances in image analysis, we tend to treat each sequence, here termed modality, in isolation. Taking advantage of the common information shared between modalities (an organ's anatomy) is beneficial for multi-modality processing and learning. However, we must overcome inherent anatomical misregistrations and disparities in signal intensity across the modalities to obtain this benefit. We present a method that offers improved segmentation accuracy of the modality of interest (over…
AnomExpert: Identifying and Selecting Anatomical Planes for Prenatal Ultrasound Anomaly Diagnosis
Medical Ultrasound Image Computing (MUSIC) Lab, School of Artificial Intelligence, Shenzhen University（医学超声图像计算（MUSIC）实验室，深圳大学人工智能学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件学院）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学大数据系统计算技术国家工程实验室）；School of Biomedical Engineering, Medical School, Shenzhen University（深圳大学医学院生物医学工程学院）；School of Biomedical Engineering and Informatics, Nanjing Medical University（南京医科大学生物医学工程与信息学院）
Biomedical Image Analysis Group, Imperial College London（伦敦帝国理工学院生物医学图像分析组）；MRC Clinical Sciences Centre (CSC)（英国医学研究委员会临床科学中心）
Incorporation of prior knowledge about organ shape and location is key to improve performance of image analysis approaches. In particular, priors can be useful in cases where images are corrupted and contain artefacts due to limitations in image acquisition. The highly constrained nature of anatomical objects can be well captured with learning based techniques. However, in most recent and promising techniques such as CNN based segmentation it is not obvious how to incorporate such prior knowledge. State-of-the-art methods operate as pixel-wise classifiers where the training objectives do not i…
Anatomy-constrained modelling of image-derived input functions in dynamic PET using multi-organ segmentation
Department of diagnostic and interventional Radiology and Neuroradiology, University Hospital Augsburg（诊断与介入放射学及神经放射学系，奥格斯堡大学医院）；Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich（计算机辅助医疗程序与增强现实，慕尼黑技术大学）；Digital Medicine, University Hospital Augsburg（数字医学，奥格斯堡大学医院）；Center of Advanced Analytics and Predictive Sciences, University of Augsburg（高级分析与预测科学中心，奥格斯堡大学）
