arXiv 2506.186582025-06-23cs.CVcs.AI

历史报告引导的双模态并发学习用于病理报告生成

Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation

Shanghai Key Laboratory of Multidimensional Information Processing, \ China Normal University, Shanghai, China

Ling Zhang, Boxiang Yun, Qingli Li, Yan Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出BiGen框架，利用历史报告引导的双模态并发学习，通过知识检索和跨模态对齐，在PathText数据集上实现病理报告生成的SOTA性能，分别提升NLP指标7.4%和分类指标19.1%。

摘要：

从全切片图像（WSIs）自动生成病理报告面临两个关键挑战：（1）视觉特征缺乏语义内容，（2）WSIs中固有的信息冗余。为解决这些问题，我们提出了一种新颖的历史报告引导的双模态并发学习框架用于病理报告生成（BiGen），模拟病理学家的诊断推理过程，包括：（1）一种知识检索机制，通过匹配高注意力补丁从预构建的医学知识库中检索与WSI相关的知识，以提供丰富的语义内容；（2）一种双模态并发学习策略，通过可学习的视觉令牌和可学习的文本令牌实例化，动态提取关键视觉特征和检索到的知识，其中权重共享层实现视觉特征与知识特征之间的跨模态对齐。我们的多模态解码器整合两种模态以生成全面的诊断报告。在PathText（BRCA）数据集上的实验证明了我们框架的优越性，在NLP指标上实现了7.4%的相对改进，在Her-2预测的分类指标上相比现有方法提升了19.1%。消融研究验证了我们所提出模块的必要性，突显了我们的方法能够提供与WSI相关的丰富语义内容并抑制WSI中的信息冗余。代码公开于https://github.com/DeepMed-Lab-ECNU/BiGen。

英文摘要：

Automated pathology report generation from Whole Slide Images (WSIs) faces two key challenges: (1) lack of semantic content in visual features and (2) inherent information redundancy in WSIs. To address these issues, we propose a novel Historical Report Guided \textbf{Bi}-modal Concurrent Learning Framework for Pathology Report \textbf{Gen}eration (BiGen) emulating pathologists' diagnostic reasoning, consisting of: (1) A knowledge retrieval mechanism to provide rich semantic content, which retrieves WSI-relevant knowledge from pre-built medical knowledge bank by matching high-attention patches and (2) A bi-modal concurrent learning strategy instantiated via a learnable visual token and a learnable textual token to dynamically extract key visual features and retrieved knowledge, where weight-shared layers enable cross-modal alignment between visual features and knowledge features. Our multi-modal decoder integrates both modals for comprehensive diagnostic reports generation. Experiments on the PathText (BRCA) dataset demonstrate our framework's superiority, achieving state-of-the-art performance with 7.4\% relative improvement in NLP metrics and 19.1\% enhancement in classification metrics for Her-2 prediction versus existing methods. Ablation studies validate the necessity of our proposed modules, highlighting our method's ability to provide WSI-relevant rich semantic content and suppress information redundancy in WSIs. Code is publicly available at https://github.com/DeepMed-Lab-ECNU/BiGen.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。