KPL: Training-Free Medical Knowledge Mining of Vision-Language Models
Zhejiang University(浙江大学);ZJU-UIUC Institute(浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院);ZJU-Angelalign R&D Center for Intelligence Healthcare(浙江大学-时代天使智能医疗研发中心);Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence(浙江省医学影像人工智能重点实验室);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局前沿人工智能研究中心);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所)
Jiaxiang Liu, Tianxiang Hu, Jiawei Du, Ruiyuan Zhang, Joey Tianyi Zhou, Zuozhu Liu
Visual Language Models such as CLIP excel in image recognition due to extensive image-text pre-training. However, applying the CLIP inference in zero-shot classification, particularly for medical image diagnosis, faces challenges due to: 1) the inadequacy of representing image classes solely with single category names; 2) the modal gap between the visual and text spaces generated by CLIP encoders. Despite attempts to enrich disease descriptions with large language models, the lack of class-specific knowledge often leads to poor performance. In addition, empirical evidence suggests that existing proxy learning methods for zero-shot image classification on natural image datasets exhibit instability when applied to medical datasets. To tackle these challenges, we introduce the Knowledge Proxy Learning (KPL) to mine knowledge from CLIP. KPL is designed to leverage CLIP's multimodal understandings for medical image classification through Text Proxy Optimization and Multimodal Proxy Learning. Specifically, KPL retrieves image-relevant knowledge descriptions from the constructed knowledge-enhanced base to enrich semantic text proxies. It then harnesses input images and these descriptions, encoded via CLIP, to stably generate multimodal proxies that boost the zero-shot classification performance. Extensive experiments conducted on both medical and natural image datasets demonstrate that KPL enables effective zero-shot image classification, outperforming all baselines. These findings highlight the great potential in this paradigm of mining knowledge from CLIP for medical image classification and broader areas.
Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis
School of Biomedical Engineering, Shanghai Jiao Tong University(上海交通大学生物医学工程学院);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能医疗科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心)
SelfMedHPM: Self Pre-training With Hard Patches Mining Masked Autoencoders For Medical Image Segmentation
1 Department of Computer Science;Engineering, Nanjing University of Aeronautics;\ , Nanjing, China 2 School of Biomedical Engineering;the Institute of Medical Robotics, Shanghai JiaoTong University, Shanghai, 200240, China
Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports
Institute of Medical Research, Northwestern Polytechnical University(西北工业大学医学研究院);School of Computing, University of Georgia(佐治亚大学计算学院);School of Life Science and Technology, University of Electronic Science and Technology of China(电子科技大学生命科学与技术学院);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Electronics and Information, Northwestern Polytechnical University(西北工业大学电子信息学院);School of Biomedical Engineering, ShanghaiTech University(上海科技大学生物医学工程学院);Lingang Laboratory(临港实验室)
Semi-Supervised Medical Image Segmentation via Knowledge Mining from Large Models
University of Cambridge(剑桥大学);Harvard Medical School(哈佛医学院);Hohai University(河海大学);Athena Research Centre(雅典娜研究中心);Tongji University(同济大学);University of Edinburgh(爱丁堡大学);Heriot-Watt University(赫瑞瓦特大学)
Lesion Harvester: Iteratively Mining Unlabeled Lesions and Hard-Negative Examples at Scale
PAII Inc.(PAII公司);Vanderbilt University(范德堡大学);Virginia Polytechnic Institute and State University(弗吉尼亚理工学院暨州立大学);Ping An Insurance (Group) Company of China, Ltd.(中国平安保险(集团)股份有限公司);University of Florida(佛罗里达大学)
DeepEDN: A Deep Learning-based Image Encryption and Decryption Network for Internet of Medical Things
University of Electronic Science and Technology of China(电子科技大学);Institute of Electronic and Information Engineering of UESTC in Guangdong(电子科技大学广东电子信息工程研究院);National Natural Science Foundation of China(国家自然科学基金委员会)
A Clinical-oriented Multi-level Contrastive Learning Method for Disease Diagnosis in Low-quality Medical Images
Northeastern University(东北大学);Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education(医学图像智能计算教育部重点实验室);National Frontiers Science Center for Industrial Intelligence and Systems Optimization(工业智能与系统优化全国前沿科学中心)
Anhui University(安徽大学);School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);First Affiliated Hospital of Anhui University of Chinese Medicine(安徽中医药大学第一附属医院);Peng Cheng Laboratory(鹏城实验室);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peking University(北京大学);National Engineering Laboratory for Video Technology(视频技术国家工程实验室)
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute(伦斯勒理工学院);Wake Forest University School of Medicine(维克森林大学医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)