arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.03754cs.CV
收藏

密集视觉引导网络用于放射学报告生成

Intensive Vision-guided Network for Radiology Report Generation

Huawei Technologies Co., Ltd.(华为技术有限公司)

Fudan Zheng, Mengfei Li, Ying Wang, Weijiang Yu, Ruixuan Wang, Zhiguang Chen, Nong Xiao, Yutong Lu

AI总结:

针对放射学报告生成中忽视多视角特征提取和多模态上下文推理的问题,提出含全局密集注意力编码器和视觉知识引导解码器的IVGN框架,在两个数据集上优于现有方法。

摘要:

自动放射学报告生成因其在医疗保健行业的巨大应用潜力而蓬勃发展。然而,现有的计算机视觉和自然语言处理方法在解决这一问题时存在两个方面的局限性。首先,在提取图像特征时,大多数方法忽略了视觉中的多视角推理,而是对医学图像的单视角结构进行建模,例如空间视角或通道视角。然而,临床医生在日常临床诊断中依赖多视角成像信息进行综合判断。其次,在生成报告时,它们忽略了利用多模态信息进行上下文推理,而专注于利用基于检索的方法进行纯文本优化。我们旨在通过提出一个更好地模拟临床医生视角并生成更准确报告的模型来解决这两个问题。鉴于上述特征提取中的局限性,我们在医学图像编码器中提出了一个全局密集注意力(GIA)模块,以模拟和整合多视角视觉感知。GIA旨在学习三种类型的视觉感知:深度视角、空间视角和像素视角。另一方面,为了解决上述报告生成中的问题,我们探索如何引入多模态信号来生成精确匹配的报告,即如何在下一个词预测中整合先前预测的词与区域感知的视觉内容。具体来说,我们设计了一个视觉知识引导解码器(VKGD),它可以自适应地考虑模型需要依赖视觉信息和先前预测文本的程度来辅助下一个词预测。因此,我们最终的密集视觉引导网络(IVGN)框架包括一个GIA引导的视觉编码器和VKGD。在两个常用数据集IU X-Ray和MIMIC-CXR上的实验表明,与其他最先进的方法相比,我们的方法具有优越的性能。

英文摘要:

Automatic radiology report generation is booming due to its huge application potential for the healthcare industry. However, existing computer vision and natural language processing approaches to tackle this problem are limited in two aspects. First, when extracting image features, most of them neglect multi-view reasoning in vision and model single-view structure of medical images, such as space-view or channel-view. However, clinicians rely on multi-view imaging information for comprehensive judgment in daily clinical diagnosis. Second, when generating reports, they overlook context reasoning with multi-modal information and focus on pure textual optimization utilizing retrieval-based methods. We aim to address these two issues by proposing a model that better simulates clinicians' perspectives and generates more accurate reports. Given the above limitation in feature extraction, we propose a Globally-intensive Attention (GIA) module in the medical image encoder to simulate and integrate multi-view vision perception. GIA aims to learn three types of vision perception: depth view, space view, and pixel view. On the other hand, to address the above problem in report generation, we explore how to involve multi-modal signals to generate precisely matched reports, i.e., how to integrate previously predicted words with region-aware visual content in next word prediction. Specifically, we design a Visual Knowledge-guided Decoder (VKGD), which can adaptively consider how much the model needs to rely on visual information and previously predicted text to assist next word prediction. Hence, our final Intensive Vision-guided Network (IVGN) framework includes a GIA-guided Visual Encoder and the VKGD. Experiments on two commonly-used datasets IU X-Ray and MIMIC-CXR demonstrate the superior ability of our method compared with other state-of-the-art approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑