BreastGPT: 面向乳腺癌临床全流程的多模态大语言模型
BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine
AI总结：
提出BreastGPT多模态大语言模型，通过构建工作流对齐的指令语料库BreastStage和双分支视觉编码器，实现乳腺癌筛查、诊断和治疗规划全流程的多模态推理，在BreastStage-Bench上取得75.66%封闭式准确率和89.92%开放式得分。
摘要：
乳腺癌仍然是女性癌症相关死亡的主要原因。其临床管理需要跨临床工作流（包括筛查、诊断和治疗规划）的多模态推理，其中每个阶段涉及不同的成像模态、任务目标和推理模式。然而，受限于数据稀缺和模型通用性，现有的医学多模态大语言模型通常仅在孤立的模态或狭窄的任务族上进行评估，限制了它们支持工作流级临床推理的能力。在这项工作中，我们首先引入了BreastStage，一个工作流对齐的乳腺影像指令语料库，包含来自5种成像模态的17个子数据集和136个任务模板的186万条指令遵循对。其保留子集BreastStage-Bench为评估乳腺癌护理连续体中的多模态推理提供了全面的基准。基于该语料库，我们提出了BreastGPT，一个统一的多模态大语言模型，配备双分支视觉编码器和概念保持的令牌压缩，以弥合标准放射学与千兆像素病理学之间的尺度差距。在BreastStage-Bench上，BreastGPT实现了75.66%的封闭式准确率和89.92%的开放式得分，在临床阶段和任务格式上均优于通用和医学专用多模态大语言模型。这些结果表明，工作流对齐的数据和跨尺度视觉建模对于临床基础的医学多模态大语言模型至关重要。所有数据、代码和模型检查点已在https://yangyy-liu.github.io/BreastGPT.io发布。
英文摘要：
Breast cancer remains a leading cause of cancer-related mortality among women. Its clinical management requires multimodal reasoning across a clinical workflow that spans \textit{screening}, \textit{diagnosis} and \textit{treatment planning}, where each stage involves distinct imaging modalities, task objectives, and reasoning patterns. However, constrained by data scarcity and model versatility, existing medical MLLMs are typically evaluated on isolated modalities or narrow task families, limiting their ability to support workflow-level clinical reasoning. In this work, we first introduce \textbf{BreastStage}, a workflow-aligned breast imaging instruction corpus comprising 1.86M instruction-following pairs curated from 17 sub-datasets across 5 imaging modalities and 136 task templates. Its held-out split, \textbf{BreastStage-Bench}, provides a comprehensive benchmark for evaluating multimodal reasoning across the breast cancer care continuum. Building on this corpus, we propose \textbf{BreastGPT}, a unified MLLM equipped with a dual-branch visual encoder and concept-preserving token compression to bridge the scale gap between standard radiology and gigapixel pathology. On BreastStage-Bench, BreastGPT achieves 75.66\% closed-ended accuracy and 89.92\% open-ended score, outperforming both general-purpose and medical-specific MLLMs across clinical stages and task formats. These results suggest that workflow-aligned data and cross-scale visual modeling are critical for clinically grounded medical MLLMs. All data, code, and model checkpoints are released at https://yangyy-liu.github.io/BreastGPT.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
