Quilt-LLaVA：通过从开源组织病理学视频中提取局部化叙事进行视觉指令调优
Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos
AI总结：
针对现有组织病理学指令调优数据集缺乏空间定位与全局视野的问题，研究团队从教育视频提取带空间锚定的问答构建Quilt-Instruct数据集，训练出支持跨块诊断推理的Quilt-LLaVA，在多项任务上显著优于现有最优模型。
摘要：
组织病理学诊断需要对全视野数字切片（WSIs）进行全局分析，要求病理学家整合来自不同WSI图像块的证据。WSI的十亿像素级规模给组织病理学多模态模型带来了挑战。训练组织病理学多模态模型需要指令调优数据集，而当前的数据集仅包含单个图像块的信息，既没有每个图像块内概念的空间定位，也缺乏对WSI的全局视野，因此不足以支撑组织病理学诊断能力。为弥补这一差距，我们提出了Quilt-Instruct，这是一个包含107131对组织病理学专属指令问答的大规模数据集，这些问答锚定在构成WSI的诊断相关图像块中。我们的数据集通过利用YouTube上的组织病理学教育视频收集，通过自动提取讲述者的光标位置实现叙事的空间定位。Quilt-Instruct通过从整个WSI中提取诊断结果和支持性事实来支持上下文推理。基于Quilt-Instruct，我们训练了Quilt-LLaVA，它能够超越给定的单个图像块进行推理，实现跨图像块的诊断推理。为评估Quilt-LLaVA，我们构建了一个由985张图像和1283个人工生成的问答组成的综合评估数据集。我们还使用公开组织病理学数据集对Quilt-LLaVA进行了全面评估，结果显示Quilt-LLaVA显著优于当前最优（SOTA）模型，相对GPT-4得分提升超过10%，在开放集和闭集视觉问答（VQA）任务上分别提升4%和9%。我们的代码、数据和模型可在quilt-llava.github.io公开获取。
英文摘要：
Diagnosis in histopathology requires a global whole slide images (WSIs) analysis, requiring pathologists to compound evidence from different WSI patches. The gigapixel scale of WSIs poses a challenge for histopathology multi-modal models. Training multi-model models for histopathology requires instruction tuning datasets, which currently contain information for individual image patches, without a spatial grounding of the concepts within each patch and without a wider view of the WSI. Therefore, they lack sufficient diagnostic capacity for histopathology. To bridge this gap, we introduce Quilt-Instruct, a large-scale dataset of 107,131 histopathology-specific instruction question/answer pairs, grounded within diagnostically relevant image patches that make up the WSI. Our dataset is collected by leveraging educational histopathology videos from YouTube, which provides spatial localization of narrations by automatically extracting the narrators' cursor positions. Quilt-Instruct supports contextual reasoning by extracting diagnosis and supporting facts from the entire WSI. Using Quilt-Instruct, we train Quilt-LLaVA, which can reason beyond the given single image patch, enabling diagnostic reasoning across patches. To evaluate Quilt-LLaVA, we propose a comprehensive evaluation dataset created from 985 images and 1283 human-generated question-answers. We also thoroughly evaluate Quilt-LLaVA using public histopathology datasets, where Quilt-LLaVA significantly outperforms SOTA by over 10% on relative GPT-4 score and 4% and 9% on open and closed set VQA. Our code, data, and model are publicly accessible at quilt-llava.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
