BIOMEDICA：一个源自科学文献的开放生物医学图文对档案、数据集及视觉-语言模型
BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature
AI总结：
本文提出开源框架BIOMEDICA，从科学文献提取超2400万图文对构建大规模生物医学数据集，并基于流式预训练发布BMCA-CLIP模型，以更少算力在40项医学任务上实现SOTA零样本分类与检索性能。
摘要：
视觉-语言模型（VLMs）的发展由大规模、多样化的多模态数据集驱动。然而，由于缺乏涵盖生物学和医学领域的带标注且公开可访问的数据集，通用生物医学VLMs的进展受到限制。现有的工作局限于狭窄的领域，缺失了科学文献中蕴含的生物医学知识的全部多样性。为了填补这一空白，我们引入了BIOMEDICA，这是一个可扩展的开源框架，用于将PubMed Central Open Access子集的全部内容提取、标注并序列化为一个易于使用、公开可访问的数据集。我们的框架生成了一个包含来自超过600万篇文章的超过2400万个独特图文对的综合档案。同时还提供了元数据和专家指导的标注。我们通过发布BMCA-CLIP展示了我们资源的实用性和可访问性，这是一套通过流式传输在BIOMEDICA数据集上连续预训练的CLIP风格模型，从而无需在本地下载27 TB的数据。平均而言，我们的模型在涵盖病理学、放射学、眼科学、皮肤病学、外科学、分子生物学、寄生虫学和细胞生物学的40项任务上实现了最先进的性能，在零样本分类中平均提升了6.56%（在皮肤病学和眼科学中分别高达29.8%和17.5%），并实现了更强的图文检索能力，而所有这些仅使用了少10倍的计算量。为了促进可复现性和协作，我们向更广泛的研究社区发布了我们的代码库和数据集。
英文摘要：
The development of vision-language models (VLMs) is driven by large-scale and diverse multimodal datasets. However, progress toward generalist biomedical VLMs is limited by the lack of annotated, publicly accessible datasets across biology and medicine. Existing efforts are restricted to narrow domains, missing the full diversity of biomedical knowledge encoded in scientific literature. To address this gap, we introduce BIOMEDICA, a scalable, open-source framework to extract, annotate, and serialize the entirety of the PubMed Central Open Access subset into an easy-to-use, publicly accessible dataset. Our framework produces a comprehensive archive with over 24 million unique image-text pairs from over 6 million articles. Metadata and expert-guided annotations are also provided. We demonstrate the utility and accessibility of our resource by releasing BMCA-CLIP, a suite of CLIP-style models continuously pre-trained on the BIOMEDICA dataset via streaming, eliminating the need to download 27 TB of data locally. On average, our models achieve state-of-the-art performance across 40 tasks - spanning pathology, radiology, ophthalmology, dermatology, surgery, molecular biology, parasitology, and cell biology - excelling in zero-shot classification with a 6.56% average improvement (as high as 29.8% and 17.5% in dermatology and ophthalmology, respectively), and stronger image-text retrieval, all while using 10x less compute. To foster reproducibility and collaboration, we release our codebase and dataset for the broader research community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
