The recent surge of foundation models in computer vision and natural language processing opens up perspectives in utilizing multi-modal clinical data to train large models with strong generalizability. Yet pathological image datasets often lack biomedical text annotation and enrichment. Guiding data-efficient image diagnosis from the use of biomedical text knowledge becomes a substantial interest. In this paper, we propose to Connect Image and Text Embeddings (CITE) to enhance pathological image classification. CITE injects text insights gained from language models pre-trained with a broad range of biomedical texts, leading to adapt foundation models towards pathological image understanding. Through extensive experiments on the PatchGastric stomach tumor pathological image dataset, we demonstrate that CITE achieves leading performance compared with various baselines especially when training data is scarce. CITE offers insights into leveraging in-domain text knowledge to reinforce data-efficient pathological image classification. Code is available at https://github.com/Yunkun-Zhang/CITE.
SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classification
School of Computer Science, Northwest University（西北大学计算机科学学院）；PuzzleLogic Pte Ltd, Singapore（新加坡PuzzleLogic公司）；Department of Electrical & Computer Engineering, National University of Singapore（新加坡国立大学电子与计算机工程系）；School of Computer Science, University of Nottingham Ningbo China（中国诺丁汉大学 Ningbo 分校计算机科学学院）
PAII Inc.（PAII公司）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；University at Buffalo（纽约州立大学布法罗分校）；Chang Gung Memorial Hospital（长庚纪念医院）；PingAn Technology（平安科技）
Mask-based annotation of medical images, especially for 3D data, is a bottleneck in developing reliable machine learning models. Using minimal-labor user interactions (UIs) to guide the annotation is promising, but challenges remain on best harmonizing the mask prediction with the UIs. To address this, we propose the user-guided domain adaptation (UGDA) framework, which uses prediction-based adversarial domain adaptation (PADA) to model the combined distribution of UIs and mask predictions. The UIs are then used as anchors to guide and align the mask prediction. Importantly, UGDA can both lear…
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
Research Unit of Computer Systems and Bioinformatics, Department of Engineering（计算机系统与生物信息学研究单位，工程系）；Research Unit of Radiation Oncology, Department of Medicine and Surgery（放射肿瘤学研究单位，医学与外科系）；Research Unit of Anatomical Pathology, Department of Medicine and Surgery（病理学研究单位，医学与外科系）
Vanderbilt University（范德堡大学）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Vanderbilt University Medical Center（范德堡大学医学中心）；Veterans Affairs Tennessee Valley Healthcare System（退伍军人事务部田纳西谷医疗系统）
Multi-instance learning (MIL) is widely used in the computer-aided interpretation of pathological Whole Slide Images (WSIs) to solve the lack of pixel-wise or patch-wise annotations. Often, this approach directly applies "natural image driven" MIL algorithms which overlook the multi-scale (i.e. pyramidal) nature of WSIs. Off-the-shelf MIL algorithms are typically deployed on a single-scale of WSIs (e.g., 20x magnification), while human pathologists usually aggregate the global and local patterns in a multi-scale manner (e.g., by zooming in and out between different magnifications). In this stu…
Multimodal Stepwise Clinically-Guided Attention Learning for Pathological Complete Response Prediction in Breast Cancer
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma（人工智能与计算机系统单位，工程系，罗马生物医学大学）；Department of Electrical Engineering and Information Technology, University of Naples Federico II（电气工程与信息科技系，那不勒斯费德里科二世大学）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University（诊断与介入系，辐射物理，生物医学工程，乌梅大学）
Diagnostic Text-guided Representation Learning in Hierarchical Classification for Pathological Whole Slide Image
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Institute of Tsinghua, Pearl River Delta（清华珠三角研究院）；Medical Optical Technology R&D Center（医疗光学技术研发中心）；The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）
提出分层病理图像分类方法 PathTree，利用二叉树结构和病理文本描述引导表示学习，在三个数据集上表现优异，为复杂 WSI 分类提供新视角。
PGVMS: A Prompt-Guided Unified Framework for Virtual Multiplex IHC Staining with Pathological Semantic Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences, Beijing, China（深圳先进技术研究院，中国科学院和中国科学院大学，北京，中国）；Department of Pathology, Sun Yat-sen University Cancer Center, State Key Laboratory of Oncology in South China, Guangzhou, China（中山大学肿瘤中心病理科，南方肿瘤临床研究中心，广州，中国）；Hong Kong Polytechnic University, Hong Kong SAR, China（香港理工大学，香港特别行政区，中国）；Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China（电子工程系，香港中文大学，香港特别行政区，中国）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China（深圳先进技术研究院，中国科学院，深圳，中国）
PA-TCNet: Pathology-Aware Temporal Calibration with Physiology-Guided Target Refinement for Cross-Subject Motor Imagery EEG Decoding in Stroke Patients
School of Artificial Intelligence, Chongqing University of Technology（重庆理工大学人工智能学院）；Chongqing Key Laboratory of Embodied Intelligence Perception and Autonomous Learning for Humanoid Robots（重庆 embodied 智能感知与人形机器人自主学习关键实验室）；Key Laboratory of Advanced Equipment Intelligence of the Chongqing Education Commission（重庆市教育委员会先进设备智能关键实验室）；School of Smart Health, Chongqing Polytechnic University of Electronic Technology（重庆电子工程职业大学智能健康学院）；Department of Language Science and Technology, The Hong Kong Polytechnic University（香港理工大学语言科学与技术系）；School of Pharmacy and Bioengineering, Chongqing University of Technology（重庆理工大学药学院与生物工程学院）；School of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Institute for AI in Medicine, University Hospital Essen（埃森大学医院人工智能医学研究所）；University of Duisburg-Essen（杜伊斯堡-埃森大学）；German Cancer Consortium (DKTK)（德国癌症联盟）
Causal Disentanglement for Robust Long-tail Medical Image Generation
School of Electrical and Information Engineering, Tianjin University, China（天津大学电气与信息工程学院）；Department of Information Engineering and Computer Science, University of Trento, Italy（特伦托大学信息工程与计算机科学系）；Fondazione Bruno Kessler, Italy（布鲁诺·克塞勒基金会）
CGSA: Class-Guided Slot-Aware Adaptation for Source-Free Object Detection
School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Department of Electrical & Computer Engineering, National University of Singapore（电气与计算机工程系，新加坡国立大学）；School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）
Enhancing Pathological VLMs with Cross-scale Reasoning
Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；PuzzleLogic Pte Ltd（PuzzleLogic私人有限公司）；Department of Pathology, Fujian Medical University Cancer Hospital & Fujian Cancer Hospital（福建医科大学附属肿瘤医院病理科暨福建省肿瘤医院）
Present domain adaptation methods usually perform explicit representation alignment by simultaneously accessing the source data and target data. However, the source data are not always available due to the privacy preserving consideration or bandwidth limitation. Source-free domain adaptation aims to solve the above problem by performing domain adaptation without accessing the source data. The adaptation paradigm is receiving more and more attention in recent years, and multiple works have been proposed for unsupervised source-free domain adaptation. However, without utilizing any supervised s…
Guided Prompt Evolution for Vision-Language Models Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Sun Yat-sen University（中山大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Kyushu University（九州大学）；Kyoto University（京都大学）；The University of Tokyo（东京大学）
Semi-supervised domain adaptation is a technique to build a classifier for a target domain by modifying a classifier in another (source) domain using many unlabeled samples and a small number of labeled samples from the target domain. In this paper, we develop a semi-supervised domain adaptation method, which has robustness to class-imbalanced situations, which are common in medical image classification tasks. For robustness, we propose a weakly-supervised clustering pipeline to obtain high-purity clusters and utilize the clusters in representation learning for domain adaptation. The proposed …
Frame-Difference Guided Dynamic Region Perception for CLIP Adaptation in Text-Video Retrieval
School of Computer Science and Technology, East China Normal University, China（上海师范大学计算机科学与技术学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
Peking University（北京大学）；South China University of Technology（华南理工大学）；National University of Singapore（新加坡国立大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
This paper studies a new, practical but challenging problem, called Class-Incremental Unsupervised Domain Adaptation (CI-UDA), where the labeled source domain contains all classes, but the classes in the unlabeled target domain increase sequentially. This problem is challenging due to two difficulties. First, source and target label sets are inconsistent at each time step, which makes it difficult to conduct accurate domain alignment. Second, previous target classes are unavailable in the current step, resulting in the forgetting of previous knowledge. To address this problem, we propose a nov…
NELVT, School of EE & CS, Peking University（北京大学电子信息与计算机科学学院NELVT）；University of Chinese Academy of Sciences（中国科学院大学）；School of ECE, Peking University（北京大学电子信息工程学院）；Pengcheng Laboratory（鹏城实验室）；Beijing Institute of Technology（北京理工大学）；Nanyang Technological University（南洋理工大学）
Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness of object detection models. However, existing UDA methods largely ignore the instantaneous data distribution during model learning, which could deteriorate the feature representation given large domain shift. In this work, we propose a Self-Guided Adaptation (SGA) model, target at aligning feature representation and transferring object detection models across domains while considering the instantaneous alignment difficulty. The core of SGA is to calculate "hardness" factors for samp…
Deep neural networks have significantly improved appearance-based gaze estimation accuracy. However, it still suffers from unsatisfactory performance when generalizing the trained model to new domains, e.g., unseen environments or persons. In this paper, we propose a plug-and-play gaze adaptation framework (PnP-GA), which is an ensemble of networks that learn collaboratively with the guidance of outliers. Since our proposed framework does not require ground-truth labels in the target domain, the existing gaze estimation networks can be directly plugged into PnP-GA and generalize the algorithms…
Nip Rumors in the Bud: Retrieval-Guided Topic-Level Adaptation for Test-Time Fake News Video Detection
University of Electronic Science and Technology of China（电子科技大学）；Aiwen Technology Co., Ltd.（爱文科技有限公司）；Intelligent Digital Media Technology Key Laboratory of Sichuan Province（四川省智能数字媒体技术重点实验室）
Domain adaptation (DA) or domain generalization (DG) for face presentation attack detection (PAD) has attracted attention recently with its robustness against unseen attack scenarios. Existing DA/DG-based PAD methods, however, have not yet fully explored the domain-specific style information that can provide knowledge regarding attack styles (e.g., materials, background, illumination and resolution). In this paper, we introduce a novel Style-Guided Domain Adaptation (SGDA) framework for inference-time adaptive PAD. Specifically, Style-Selective Normalization (SSN) is proposed to explore the do…
Northeastern University（东北大学）；University of the Chinese Academy of Sciences（中国科学院大学）；China Medical University（中国医科大学）
The deep neural network is a research hotspot for histopathological image analysis, which can improve the efficiency and accuracy of diagnosis for pathologists or be used for disease screening. The whole slide pathological image can reach one gigapixel and contains abundant tissue feature information, which needs to be divided into a lot of patches in the training and inference stages. This will lead to a long convergence time and large memory consumption. Furthermore, well-annotated data sets are also in short supply in the field of digital pathology. Inspired by the pathologist's clinical di…
College of Computer Science and Technology, Huaqiao University（华侨大学计算机科学与技术学院）
Semantic segmentation is an essential step for electron microscopy (EM) image analysis. Although supervised models have achieved significant progress, the need for labor intensive pixel-wise annotation is a major limitation. To complicate matters further, supervised learning models may not generalize well on a novel dataset due to domain shift. In this study, we introduce an adversarial-prediction guided multi-task network to learn the adaptation of a well-trained model for use on a novel unlabeled target domain. Since no label is available on target domain, we learn an encoding representation…
Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
School of Engineering Science, Simon Fraser University（西蒙弗雷泽大学工程科学学院）；School of Information Science and Technology, Eastern Institute of Technology（东方理工学院信息科学与技术学院）
Domain Adaptive Multiple Instance Learning for Instance-level Prediction of Pathological Images
The University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Kyoto University（京都大学）；The Japanese Society of Pathology（日本病理学会）；Tokyo Medical and Dental University（东京医科齿科大学）；National Institute of Informatics（国立信息学研究所）
