PathAlign：一种用于组织病理学全切片图像的视觉-语言模型
PathAlign: A vision-language model for whole slide images in histopathology
AI总结：
本文提出PathAlign，基于BLIP-2框架利用超35万对WSI与病理报告文本训练视觉-语言模型，实现文本/图像检索、报告生成及切片级分诊，病理学家评估显示78%的生成文本准确无临床显著错误。
摘要：
组织病理学图像的显微解读是许多重要诊断和治疗决策的基础。尽管视觉-语言建模的进展为分析此类图像带来了新的机遇，但全切片图像（WSIs）的千兆像素级尺寸带来了独特的挑战。此外，病理报告同时突出小区域的关键发现，又汇总跨多张切片的解读，这往往使构建稳健的图像-文本对变得困难。因此，病理报告在计算病理学中仍然是一个很大程度上未被利用的监督来源，大多数工作依赖于感兴趣区域标注或切片块级别的自监督。在这项工作中，我们基于BLIP-2框架开发了一个视觉-语言模型，使用与病理报告中精选文本配对的WSIs。这使得利用共享图像-文本嵌入空间的应用成为可能，例如用于查找感兴趣病例的文本或图像检索，以及将WSI编码器与冻结的大语言模型（LLM）集成，以实现基于WSI的生成式文本能力，如报告生成或AI在环交互。我们利用一个去标识化的数据集，包含超过350,000个WSIs和诊断文本对，涵盖广泛的诊断、操作类型和组织类型。我们展示了病理学家对使用WSI嵌入进行文本生成和文本检索的评估，以及WSI分类和工作流程优先级排序（切片级分诊）的结果。病理学家对模型为WSIs生成的文本进行评分，平均78%的WSIs被评为准确，没有临床显著错误或遗漏。这项工作展示了语言对齐的WSI嵌入令人兴奋的潜在能力。
英文摘要：
Microscopic interpretation of histopathology images underlies many important diagnostic and treatment decisions. While advances in vision-language modeling raise new opportunities for analysis of such images, the gigapixel-scale size of whole slide images (WSIs) introduces unique challenges. Additionally, pathology reports simultaneously highlight key findings from small regions while also aggregating interpretation across multiple slides, often making it difficult to create robust image-text pairs. As such, pathology reports remain a largely untapped source of supervision in computational pathology, with most efforts relying on region-of-interest annotations or self-supervision at the patch-level. In this work, we develop a vision-language model based on the BLIP-2 framework using WSIs paired with curated text from pathology reports. This enables applications utilizing a shared image-text embedding space, such as text or image retrieval for finding cases of interest, as well as integration of the WSI encoder with a frozen large language model (LLM) for WSI-based generative text capabilities such as report generation or AI-in-the-loop interactions. We utilize a de-identified dataset of over 350,000 WSIs and diagnostic text pairs, spanning a wide range of diagnoses, procedure types, and tissue types. We present pathologist evaluation of text generation and text retrieval using WSI embeddings, as well as results for WSI classification and workflow prioritization (slide-level triaging). Model-generated text for WSIs was rated by pathologists as accurate, without clinically significant error or omission, for 78% of WSIs on average. This work demonstrates exciting potential capabilities for language-aligned WSI embeddings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
