Deep neural networks are increasingly used in medical imaging for tasks such as pathological classification, but they face challenges due to the scarcity of high-quality, expert-labeled training data. Recent efforts have utilized pre-trained contrastive image-text models like CLIP, adapting them for medical use by fine-tuning the model with chest X-ray images and corresponding reports for zero-shot pathology classification, thus eliminating the need for pathology-specific annotations. However, most studies continue to use the same contrastive learning objectives as in the general domain, overlooking the multi-labeled nature of medical image-report pairs. In this paper, we propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. We aim to improve the performance of zero-shot pathology classification without relying on external knowledge. Our method can be applied to any pre-trained contrastive image-text encoder and easily transferred to out-of-domain datasets without further training, as it does not use external data. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models, with an average macro AUROC increase of 4.3%. Additionally, our method outperforms the state-of-the-art and marginally surpasses board-certified radiologists in zero-shot classification for the five competition pathologies in the CheXpert dataset.
Emory University（埃默里大学）；Indiana University Purdue University-Indianapolis（印第安纳大学与普渡大学印第安纳波利斯联合分校）
Deep learning approaches applied to medical imaging have reached near-human or better-than-human performance on many diagnostic tasks. For instance, the CheXpert competition on detecting pathologies in chest x-rays has shown excellent multi-class classification performance. However, training and validating deep learning models require extensive collections of images and still produce false inferences, as identified by a human-in-the-loop. In this paper, we introduce a practical approach to improve the predictions of a pre-trained model through Few-Shot Learning (FSL). After training and valida…
Dino-NestedUNet: Unlocking Foundation Vision Encoders for Pathology Tumor Bulk Segmentation via Dense Decoding
Department of Pathology, College of Medicine, The Ohio State University Wexner Medical Center, Columbus, OH, USA（病理学系，医学学院，俄亥俄州立大学韦克斯纳医学中心，哥伦布，OH，USA）；Department of Biomedical Engineering, The Ohio State University, Columbus, OH, USA（生物医学工程系，俄亥俄州立大学，哥伦布，OH，USA）
Tencent AI Lab（腾讯AI实验室）；University of Connecticut（康涅狄格大学）；The Fourth Hospital of Hebei Medical University（河北医科大学第四医院）；West China Hospital, Sichuan University（四川大学华西医院）
Artificial Intelligence (AI)-powered pathology is a revolutionary step in the world of digital pathology and shows great promise to increase both diagnosis accuracy and efficiency. However, defocus and motion blur can obscure tissue or cell characteristics hence compromising AI algorithms'accuracy and robustness in analyzing the images. In this paper, we demonstrate a deep-learning-based approach that can alleviate the defocus and motion blur of a microscopic image and output a sharper and cleaner image with retrieved fine details without prior knowledge of the blur type, blur extent and patho…
DeepAf: One-Shot Spatiospectral Auto-Focus Model for Digital Pathology
Chair for Computer Aided Medical Procedures (CAMP), TU Munich, Germany（计算机辅助医学程序研究所（CAMP），慕尼黑技术大学，德国）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Southern Taiwan University of Science and Technology, Taiwan（台湾科技大学）；Harvard Data Science Initiative（哈佛大学数据科学计划）；Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA, USA（自然与人工智能研究学院，哈佛大学，剑桥，马萨诸塞州，美国）
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）；McMaster University（麦克马斯特大学）；Shanghai Jiao Tong University（上海交通大学）
Whole slide imaging (WSI) is an emerging technology for digital pathology. The process of autofocusing is the main influence of the performance of WSI. Traditional autofocusing methods either are time-consuming due to repetitive mechanical motions, or require additional hardware and thus are not compatible to current WSI systems. In this paper, we propose the concept of \textit{virtual autofocusing}, which does not rely on mechanical adjustment to conduct refocusing but instead recovers in-focus images in an offline learning-based manner. With the initial focal position, we only perform two-sh…
Multimodal Whole Slide Foundation Model for Pathology
Mass General Brigham, Harvard Medical School（哈佛医学院，麻省总医院布里格姆医疗系统）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Broad Institute of Harvard and MIT（哈佛与麻省理工学院博德研究所）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑中心——德国环境与健康研究中心）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology（麻省理工学院哈佛-麻省卫生科学与技术分部）；Department of Pathology, Pusan National University（釜山大学病理学系）；Department of Pathology and Laboratory Medicine, Emory University School of Medicine（埃默里大学医学院病理学与检验医学系）；Harvard Data Science Initiative, Harvard University（哈佛大学哈佛数据科学计划）；Department of Systems Biology, Harvard Medical School（哈佛医学院系统生物学系）；Department of Preventive Medicine, Graduate School of Medicine, The University of Tokyo（东京大学医学部预防医学系）
GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment Analysis
Microsoft Research（微软研究院）；Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Providence Genomics（普罗维登斯基因组学公司）；Earle A. Chiles Research Institute, Providence Cancer Institute（普罗维登斯癌症研究所厄尔·A·奇尔斯研究所）；Providence Research Network（普罗维登斯研究网络）
Unsupervised learning enables modeling complex images without the need for annotations. The representation learned by such models can facilitate any subsequent analysis of large image datasets. However, some generative factors that cause irrelevant variations in images can potentially get entangled in such a learned representation causing the risk of negatively affecting any subsequent use. The orientation of imaged objects, for instance, is often arbitrary/irrelevant, thus it can be desired to learn a representation in which the orientation information is disentangled from all other factors. …
As many algorithms depend on a suitable representation of data, learning unique features is considered a crucial task. Although supervised techniques using deep neural networks have boosted the performance of representation learning, the need for a large set of labeled data limits the application of such methods. As an example, high-quality delineations of regions of interest in the field of pathology is a tedious and time-consuming task due to the large image dimensions. In this work, we explored the performance of a deep neural network and triplet loss in the area of representation learning.…
Leveraging Weak Supervision for Cell Localization in Digital Pathology Using Multitask Learning and Consistency Loss
Koc University（科克大学）；Basaksehir Cam and Sakura City Hospital（巴沙克谢伊尔卡姆与樱花城市医院）；Sisli Hamidiye Etfal Health Application and Research Center（锡什利哈米迪耶埃特法尔健康应用与研究中心）；Koc University School of Medicine（科克大学医学院）
Prior-Anchored Debiasing for Long-Tailed Multi-Organ Pathology Report Generation
City University of Hong Kong, Hong Kong SAR（香港城市大学）；University of Nottingham, United Kingdom（诺丁汉大学）；University of Massachusetts Boston, United States（马萨诸塞大学波士顿分校）
Integrating Pathology and CT Imaging for Personalized Recurrence Risk Prediction in Renal Cancer
Department of Radiology, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, The Netherlands（放射科，Antoni van Leeuwenhoek荷兰癌症研究所，阿姆斯特丹，荷兰）；University of Amsterdam（阿姆斯特丹大学）；AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University（AI技术生命，信息与计算科学系，生物学系，乌得勒支大学）；GROW Oncology, Maastricht University（GROW肿瘤学，马斯特里赫特大学）；Department of Oncology, University of Cambridge（肿瘤科，剑桥大学）；Cancer Research UK Cambridge Centre, University of Cambridge（英国癌症研究会剑桥中心，剑桥大学）；Early Cancer Institute, University of Cambridge（早期癌症研究所，剑桥大学）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院 NHS 基础信托）
We present ARCH, a computational pathology (CP) multiple instance captioning dataset to facilitate dense supervision of CP tasks. Existing CP datasets focus on narrow tasks; ARCH on the other hand contains dense diagnostic and morphological descriptions for a range of stains, tissue types and pathologies. Using intrinsic dimensionality estimation, we show that ARCH is the only CP dataset to (ARCH-)rival its computer vision analog MS-COCO Captions. We conjecture that an encoder pre-trained on dense image captions learns transferable representations for most CP tasks. We support the conjecture w…
SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Space
Biomedical AI Research Lab, University of California, Los Angeles（生物医学人工智能研究实验室，加州大学洛杉矶分校）；Department of Pathology, University of California, Los Angeles（病理学系，加州大学洛杉矶分校）；Department of Radiology, University of California, Los Angeles（放射学系，加州大学洛杉矶分校）；Department of Bioengineering, University of California, Los Angeles（生物工程系，加州大学洛杉矶分校）；UCLA Medical Informatics Home Area, University of California, Los Angeles（UCLA医学信息学家庭区域，加州大学洛杉矶分校）；Department of Computational Medicine, University of California, Los Angeles（计算医学系，加州大学洛杉矶分校）
Pohang University of Science and Technology (POSTECH)（浦项科技大学）
Zero-shot slot filling has received considerable attention to cope with the problem of limited available data for the target domain. One of the important factors in zero-shot learning is to make the model learn generalized and reliable representations. For this purpose, we present mcBERT, which stands for momentum contrastive learning with BERT, to develop a robust zero-shot slot filling model. mcBERT uses BERT to initialize the two encoders, the query encoder and key encoder, and is trained by applying momentum contrastive learning. Our experimental results on the SNIPS benchmark show that mc…
University of Toronto（多伦多大学）；Vector Institute；University of Tokyo（东京大学）
A crucial challenge in image-based modeling of biomedical data is to identify trends and features that separate normality and pathology. In many cases, the morphology of the imaged object exhibits continuous change as it deviates from normality, and thus a generative model can be trained to model this morphological continuum. Moreover, given side information that correlates to certain trend in morphological change, a latent variable model can be regularized such that its latent representation reflects this side information. In this work, we use the Wasserstein Auto-encoder to model this pathol…
Can We Simplify Slide-level Fine-tuning of Pathology Foundation Models?
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Oxford（牛津大学）；The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）；University of Washington（华盛顿大学）
FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology
Giessen University（吉森大学）；University Medicine Gottingen（哥廷根大学医学中心）；Technical University of Denmark（丹麦技术大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Max Planck Institute for Biology of Ageing（马克斯·普朗克衰老生物学研究所）；National Taiwan University（国立台湾大学）；VNU University of Medicine and Pharmacy（越南国立大学医学与药学大学）；University of Medicine and Pharmacy, Hue University（顺化大学医学与药学大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；University of Stuttgart（斯图加特大学）
Transductive methods always outperform inductive methods in few-shot image classification scenarios. However, the existing few-shot methods contain a latent condition: the number of samples in each class is the same, which may be unrealistic. To cope with those cases where the query shots of each class are nonuniform (i.e. nonuniform few-shot learning), we propose a Task-Prior Conditional Variational Auto-Encoder model named TP-VAE, conditioned on support shots and constrained by a task-level prior regularization. Our method obtains high performance in the more challenging nonuniform few-shot …
PathCoT: Chain-of-Thought Prompting for Zero-shot Pathology Visual Reasoning
The College of Artificial Intelligence, Nanjing University of Aeronautics（人工智能学院，南京航空航天大学）；Astronautics The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（航空航天脑机智能技术重点实验室，教育部）
Prompt, Plan, Extract: Zero-Shot Agentic LLMs Workflows for Lung Pathology Extraction from Clinical Narratives
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida（健康结果与生物医学信息学系，医学院，佛罗里达大学）；Division of Pulmonary, Critical Care and Sleep Medicine, Department of Medicine, College of Medicine, University of Florida（呼吸科、重症医学科和睡眠医学科，医学系，医学院，佛罗里达大学）；College of Nursing, Florida State University（护理学院，佛罗里达州立大学）
Evaluating Computational Pathology Foundation Models for Prostate Cancer Grading under Distribution Shifts
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet（卡罗林斯卡研究所医学流行病学与生物统计学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）
A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification
Bakar Computational Health Sciences Institute, University of California, San Francisco（加州大学旧金山分校巴卡尔计算健康科学研究所）；Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco（加州大学旧金山分校海伦·迪勒家族综合癌症中心）；University of California, Berkeley（加州大学伯克利分校）；Center for Data-driven Insights and Innovation, University of California, Office of the President（加州大学校长办公室数据驱动洞察与创新中心）；Department of Pediatrics, University of California, San Francisco（加州大学旧金山分校儿科系）
Most of the existing artificial neural networks(ANNs) fail to learn continually due to catastrophic forgetting, while humans can do the same by maintaining previous tasks' performances. Although storing all the previous data can alleviate the problem, it takes a large memory, infeasible in real-world utilization. We propose a continual zero-shot learning model(A-CZSL) that is more suitable in real-case scenarios to address the issue that can learn sequentially and distinguish classes the model has not seen during training. Further, to enhance the reliability, we develop A-CZSL for a single hea…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University Medicine Essen（埃森大学医学院）
When reading images, radiologists generate text reports describing the findings therein. Current state-of-the-art computer-aided diagnosis tools utilize a fixed set of predefined categories automatically extracted from these medical reports for training. This form of supervision limits the potential usage of models as they are unable to pick up on anomalies outside of their predefined set, thus, making it a necessity to retrain the classifier with additional data when faced with novel classes. In contrast, we investigate direct text supervision to break away from this closed set assumption. By…
We propose an end-to-end neural model for zero-shot abstractive text summarization of paragraphs, and introduce a benchmark task, ROCSumm, based on ROCStories, a subset for which we collected human summaries. In this task, five-sentence stories (paragraphs) are summarized with one sentence, using human summaries only for evaluation. We show results for extractive and human baselines to demonstrate a large abstractive gap in performance. Our model, SummAE, consists of a denoising auto-encoder that embeds sentences and paragraphs in a common space, from which either can be decoded. Summaries for…
University of Augsburg（奥格斯堡大学）；University of Tokyo（东京大学）；Imperial College（帝国理工学院）
In this work, we explore a novel few-shot personalisation architecture for emotional vocalisation prediction. The core contribution is an `enrolment' encoder which utilises two unlabelled samples of the target speaker to adjust the output of the emotion encoder; the adjustment is based on dot-product attention, thus effectively functioning as a form of `soft' feature selection. The emotion and enrolment encoders are based on two standard audio architectures: CNN14 and CNN10. The two encoders are further guided to forget or learn auxiliary emotion and/or speaker information. Our best approach a…
