无需标注的组织病理学视觉语言模型轻松特化
Effortless Vision-Language Model Specialization in Histopathology without Annotation
AI总结：
本文提出一种无需标注的VLM适配方法，通过在领域相关图文对上持续预训练，显著提升了组织病理学下游任务的零样本与少样本性能，效果媲美少样本方法且免除人工标注。
摘要：
组织病理学中视觉语言模型（VLMs）的最新进展，如CONCH和QuiltNet，已展现出在各种任务中令人印象深刻的零样本分类能力。然而，其通用设计可能导致在特定下游应用中性能次优。尽管有监督微调方法解决了这一问题，但它们需要人工标注样本进行适配。本文通过在从现有数据库提取的领域和任务相关图文对上进行持续预训练，研究了VLMs的无标注适配。我们在两个VLMs（CONCH和QuiltNet）和三个下游任务上的实验表明，这些图文对显著提升了零样本和少样本性能。值得注意的是，随着训练规模增大，持续预训练在消除人工标注的同时，其性能可媲美少样本方法。其有效性、任务无关的设计及无标注工作流，使其成为将VLMs适配到新组织病理学任务的有前景的途径。代码可在https://github.com/DeepMicroscopy/Annotation-free-VLM-specialization获取。
英文摘要：
Recent advances in Vision-Language Models (VLMs) in histopathology, such as CONCH and QuiltNet, have demonstrated impressive zero-shot classification capabilities across various tasks. However, their general-purpose design may lead to suboptimal performance in specific downstream applications. While supervised fine-tuning methods address this issue, they require manually labeled samples for adaptation. This paper investigates annotation-free adaptation of VLMs through continued pretraining on domain- and task-relevant image-caption pairs extracted from existing databases. Our experiments on two VLMs, CONCH and QuiltNet, across three downstream tasks reveal that these pairs substantially enhance both zero-shot and few-shot performance. Notably, with larger training sizes, continued pretraining matches the performance of few-shot methods while eliminating manual labeling. Its effectiveness, task-agnostic design, and annotation-free workflow make it a promising pathway for adapting VLMs to new histopathology tasks. Code is available at https://github.com/DeepMicroscopy/Annotation-free-VLM-specialization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
