Unsupervised learning algorithms (e.g., self-supervised learning, auto-encoder, contrastive learning) allow deep learning models to learn effective image representations from large-scale unlabeled data. In medical image analysis, even unannotated data can be difficult to obtain for individual labs. Fortunately, national-level efforts have been made to provide efficient access to obtain biomedical image data from previous scientific publications. For instance, NIH has launched the Open-i search engine that provides a large-scale image database with free access. However, the images in scientific publications consist of a considerable amount of compound figures with subplots. To extract and curate individual subplots, many different compound figure separation approaches have been developed, especially with the recent advances in deep learning. However, previous approaches typically required resource extensive bounding box annotation to train detection models. In this paper, we propose a simple compound figure separation (SimCFS) framework that uses weak classification annotations from individual images. Our technical contribution is three-fold: (1) we introduce a new side loss that is designed for compound figure separation; (2) we introduce an intra-class image augmentation method to simulate hard cases; (3) the proposed framework enables an efficient deployment to new classes of images, without requiring resource extensive bounding box annotations. From the results, the SimCFS achieved a new state-of-the-art performance on the ImageCLEF 2016 Compound Figure Separation Database. The source code of SimCFS is made publicly available at https://github.com/hrlblab/ImageSeperation.
COVID-19-CT-CXR: a freely accessible and weakly labeled chest X-ray and CT image collection on COVID-19 from biomedical literature
National Center for Biotechnology Information (NCBI)(美国国家生物技术信息中心);National Library of Medicine (NLM)(美国国家医学图书馆);National Institutes of Health (NIH)(美国国立卫生研究院);Weill Cornell Medicine(威尔康奈尔医学院);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
对全球健康的最新威胁是COVID-19疫情。尽管存在胸部X射线(CXR)和计算机断层扫描(CT)扫描的大型数据集,但由于患者隐私,目前很少有COVID-19图像收集。与此同时,生物医学文献中与COVID-19相关的文章迅速增长。在这里,我们介绍了COVID-19-CT-CXR,这是一个COVID-19 CXR和CT图像的公共数据库,这些图像是从PubMed Central Open Access(PMC-OA)子集的COVID-19相关文章中自动提取的。我们提取了文章中的图形、相关说明和相关图形描述,并将复合图形分成子图形。我们还设计了一个深度学习模型,将它们与其他图形类型区分开来,并对其进行相应的分类。最终数据库包括1,327张CT和263张CXR图像(截至2020年5月9日)及其相关文本。为了证明COVID-19-CT-CXR的效用,我们进行了四个案例研究。(1)我们表明,COVID-19-CT-CXR在用作额外的训练数据时,能够有助于提高COVID-19和非COVID-19 CT分类的DL性能。(2)我们收集了流感的CT图像,并训练了DL基线,以区分COVID-19、流感或CT上的正常或其他类型疾病的诊断。(3)我们从非COVID-19 CXR中训练了一个无监督的单类分类器,并执行异常检测来检测COVID-19 CXR。(4)从文本挖掘的标题和图形描述中,我们比较了COVID…
From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature
Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT 06510, USA(耶鲁医学院生物医学信息学与数据科学系,耶鲁大学,新 Haven,CT 06510,USA);School of Medicine, University of Puerto Rico, San Juan, PR 00921, USA(波多黎各大学医学院,波多黎各,San Juan,PR 00921,USA)
MedICaT: A Dataset of Medical Images, Captions, and Textual References
University of Washington(华盛顿大学);Tel Aviv University(特拉维夫大学);Swedish Medical Group(瑞典医疗集团);University of California, Irvine(加州大学欧文分校);Allen Institute for AI(艾伦人工智能研究所)
From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France(斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、斯特拉斯堡,法国);Technical University of Munich(慕尼黑技术大学);Stanford University(斯坦福大学);DSAI, The Hong Kong Polytechnic University(香港理工大学DSAI实验室);University of British Columbia(不列颠哥伦比亚大学);Munich Center for Machine Learning(慕尼黑机器学习中心);IHU Strasbourg, Strasbourg(斯特拉斯堡IHU医院,斯特拉斯堡);Vector Institute for AI(人工智能向量研究所);Yale University(耶鲁大学)
Multi-Compound Transformer for Accurate Biomedical Image Segmentation
The University of Hong Kong(香港大学);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)深圳大数据研究院);SenseTime Research(商汤科技研究院)
Exploring scalable medical image encoders beyond text supervision
Health Futures, Microsoft Research(微软研究院健康未来部门);Microsoft Health and Life Sciences(微软健康与生命科学部门);Department of Radiology, University of Cambridge(剑桥大学放射学系);Cambridge University Hospitals NHS Foundation Trust(剑桥大学医院NHS基金会信托);Microsoft Azure AI(微软Azure人工智能部门)
MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature
Mehta Family School of Data Science and Artificial Intelligence(梅塔家庭数据科学与人工智能学院);Indian Institute of Technology Roorkee(印度理工学院罗奥克学院);Department of Metallurgical and Materials Engineering(冶金与材料工程系)
PlotPick: AI-powered batch extraction of numerical data from scientific figures
Copenhagen Research Centre for Biological and Precision Psychiatry(哥本哈根生物与精准精神病学研究中心);Mental Health Centre Copenhagen(哥本哈根心理健康中心);Copenhagen University Hospital(哥本哈根大学医院)
MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
Yale University(耶鲁大学);Korea University(韩国大学);Washington University in St. Louis(圣路易斯华盛顿大学);The University of Queensland(昆士兰大学);The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);University of Washington(华盛顿大学);Microsoft Research(微软研究院)