基于交叉注意力融合的医学视觉问答分层建模
Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
AI总结：
针对分层医学视觉问答中层级建模不完善和跨模态融合隐式学习不足的问题，提出HiCA-VQA方法，通过分层提示、分层答案解码器和图像为查询的交叉注意力融合，在Rad-Restruct基准上取得了优于现有方法的细粒度问答性能。
摘要：
医学视觉问答（Med-VQA）利用医学图像回答临床问题，以辅助诊断。设计MedVQA系统对于辅助临床诊断和提高诊断准确性具有重要意义。在此基础上，分层医学VQA通过将医学问题组织为分层结构并进行层级特定预测来扩展医学VQA，以处理细粒度区分。近年来，许多研究提出了分层MedVQA任务并建立了数据集，但仍存在若干问题：（1）不完善的分层建模导致问题层级之间区分度差，造成跨层级语义碎片化。（2）在基于Transformer的跨模态自注意力融合方法中过度依赖隐式学习，掩盖了医学场景中关键的局部语义相关性。为解决这些问题，本研究提出了一种HiCA-VQA方法，包括两个模块：用于细粒度医学问题的分层提示（Hierarchical Prompting）和分层答案解码器（Hierarchical Answer Decoders）。分层提示模块将分层文本提示与图像特征预先对齐，以引导模型根据问题类型关注特定图像区域，而分层解码器对不同层级的问题分别进行预测，以提高不同粒度下的准确性。该框架还融合了一个交叉注意力融合模块，其中图像作为查询，文本作为键值对。在Rad-Restruct基准上的实验表明，HiCA-VQA框架在回答分层细粒度问题方面优于现有最先进方法。本研究为分层视觉问答系统提供了一条有效途径，推动了医学图像理解的发展。
英文摘要：
Medical Visual Question Answering (Med-VQA) answers clinical questions using medical images, aiding diagnosis. Designing the MedVQA system holds profound importance in assisting clinical diagnosis and enhancing diagnostic accuracy. Building upon this foundation, Hierarchical Medical VQA extends Medical VQA by organizing medical questions into a hierarchical structure and making level-specific predictions to handle fine-grained distinctions. Recently, many studies have proposed hierarchical MedVQA tasks and established datasets, However, several issues still remain: (1) imperfect hierarchical modeling leads to poor differentiation between question levels causing semantic fragmentation across hierarchies. (2) Excessive reliance on implicit learning in Transformer-based cross-modal self-attention fusion methods, which obscures crucial local semantic correlations in medical scenarios. To address these issues, this study proposes a HiCA-VQA method, including two modules: Hierarchical Prompting for fine-grained medical questions and Hierarchical Answer Decoders. The hierarchical prompting module pre-aligns hierarchical text prompts with image features to guide the model in focusing on specific image regions according to question types, while the hierarchical decoder performs separate predictions for questions at different levels to improve accuracy across granularities. The framework also incorporates a cross-attention fusion module where images serve as queries and text as key-value pairs. Experiments on the Rad-Restruct benchmark demonstrate that the HiCA-VQA framework better outperforms existing state-of-the-art methods in answering hierarchical fine-grained questions. This study provides an effective pathway for hierarchical visual question answering systems, advancing medical image understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
