arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.02476cs.CVcs.AI
收藏

BioD2C:一种用于生物医学VQA的双层语义一致性约束框架

BioD2C: A Dual-level Semantic Consistency Constraint Framework for Biomedical VQA

Shandong University(山东大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Institute of Deep Perception Technology, JITRI(JITRI深度感知技术研究院)

Zhengyang Ji, Shang Gao, Li Liu, Yifan Jia, Yutao Yue

AI总结:

针对生物医学VQA多模态语义对齐欠佳问题,提出双层语义一致性约束框架BioD2C,通过图像-文本融合与跨模态软语义损失实现特征与模型双重对齐,并构建新数据集BioVGQ,在多个下游数据集上达到SOTA。

摘要:

生物医学视觉问答(VQA)已被广泛研究,并在辅助医疗诊断等领域展现出显著的应用价值与潜力。尽管取得了成功,但当前的生物医学VQA模型仅在大语言模型(LLMs)内部的模型层面进行多模态信息交互,导致在处理复杂任务时多模态语义对齐效果欠佳。为解决此问题,我们提出了BioD2C:一种用于生物医学VQA的新型双层语义一致性约束框架,该框架在模型和特征两个层面实现双层语义交互对齐,使模型能够基于问题自适应地学习视觉特征。具体而言,我们首先通过图像-文本融合机制将文本特征整合到视觉特征中,作为特征层面的语义交互,从而获得以给定文本为条件的视觉特征;随后引入一种基于文本队列的跨模态软语义损失函数,以进一步将图像语义与问题语义对齐。具体来说,在本项工作中,我们构建了一个新数据集BioVGQ,通过过滤人工修改的图像并使问答对与多模态上下文对齐,来解决先前数据集中固有的偏差,并在该数据集上训练了我们的模型。大量实验结果表明,BioD2C在多个下游数据集上取得了最先进的(SOTA)性能,展示了其鲁棒性、泛化能力以及推进生物医学VQA研究的潜力。

英文摘要:

Biomedical visual question answering (VQA) has been widely studied and has demonstrated significant application value and potential in fields such as assistive medical diagnosis. Despite their success, current biomedical VQA models perform multimodal information interaction only at the model level within large language models (LLMs), leading to suboptimal multimodal semantic alignment when dealing with complex tasks. To address this issue, we propose BioD2C: a novel Dual-level Semantic Consistency Constraint Framework for Biomedical VQA, which achieves dual-level semantic interaction alignment at both the model and feature levels, enabling the model to adaptively learn visual features based on the question. Specifically, we firstly integrate textual features into visual features via an image-text fusion mechanism as feature-level semantic interaction, obtaining visual features conditioned on the given text; and then introduce a text-queue-based cross-modal soft semantic loss function to further align the image semantics with the question semantics. Specifically, in this work, we establish a new dataset, BioVGQ, to address inherent biases in prior datasets by filtering manually-altered images and aligning question-answer pairs with multimodal context, and train our model on this dataset. Extensive experimental results demonstrate that BioD2C achieves state-of-the-art (SOTA) performance across multiple downstream datasets, showcasing its robustness, generalizability, and potential to advance biomedical VQA research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑