SlideChat：面向全切片病理图像理解的大型视觉语言助手
SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding
AI总结：
针对现有多模态大模型仅聚焦病理图像斑块级分析、缺乏全切片上下文的问题，本文提出首个支持十亿像素全切片病理图像理解的视觉语言助手SlideChat，构建了最大规模WSI指令数据集SlideInstruction与多模态基准SlideBench，在22项任务中18项达SOTA。
摘要：
尽管多模态大语言模型（MLLMs）在计算病理学领域已取得进展，但它们仍受限于主要聚焦于斑块级分析的问题，缺失了全切片层面的关键上下文信息。大规模指令数据集的匮乏以及全切片图像（WSIs）的十亿像素级规模，构成了显著的开发挑战。本文提出SlideChat，这是首个能够理解十亿像素级全切片图像的视觉语言助手，在多样的病理学场景中展现出出色的多模态对话能力，可响应复杂指令。为支撑其开发，我们构建了SlideInstruction——目前最大的WSI指令遵循数据集，包含4200条WSI字幕和17.6万对多类别视觉问答（VQA）对。此外，我们提出了多模态基准SlideBench，涵盖字幕生成和VQA任务，用于评估SlideChat在显微镜检查、诊断等不同临床场景下的能力。与通用和专用MLLMs相比，SlideChat表现出卓越性能，在22项任务中的18项上达到了当前最优水平。例如，它在SlideBench-VQA（TCGA）上取得了81.17%的整体准确率，在SlideBench-VQA（BCNB）上达到54.15%。我们的代码、数据和模型已在https://uni-medical.github.io/SlideChat.github.io 公开发布。
英文摘要：
Despite the progress made by multimodal large language models (MLLMs) in computational pathology, they remain limited by a predominant focus on patch-level analysis, missing essential contextual information at the whole-slide level. The lack of large-scale instruction datasets and the gigapixel scale of whole slide images (WSIs) pose significant developmental challenges. In this paper, we present SlideChat, the first vision-language assistant capable of understanding gigapixel whole-slide images, exhibiting excellent multimodal conversational capability and response complex instruction across diverse pathology scenarios. To support its development, we created SlideInstruction, the largest instruction-following dataset for WSIs consisting of 4.2K WSI captions and 176K VQA pairs with multiple categories. Furthermore, we propose SlideBench, a multimodal benchmark that incorporates captioning and VQA tasks to assess SlideChat's capabilities in varied clinical settings such as microscopy, diagnosis. Compared to both general and specialized MLLMs, SlideChat exhibits exceptional capabilities achieving state-of-the-art performance on 18 of 22 tasks. For example, it achieved an overall accuracy of 81.17% on SlideBench-VQA (TCGA), and 54.15% on SlideBench-VQA (BCNB). Our code, data, and model is publicly accessible at https://uni-medical.github.io/SlideChat.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
