基于VisionLLM的多模态融合网络用于声门癌早期检测
VisionLLM-based Multimodal Fusion Network for Glottic Carcinoma Early Detection
AI总结:
针对声门癌与声带发育不良形态相似导致检测准确率不佳的问题,提出基于VisionLLM的多模态融合网络MMGC-Net,通过融合图像与文本特征提升早期检测性能。
摘要:
声门癌的早期检测对于改善患者预后至关重要,因为它能够实现及时干预、保留发声功能,并显著降低肿瘤进展和转移的风险。然而,声门癌与声带发育不良在形态上的相似性导致检测准确率不佳。为了解决这个问题,我们提出了一种基于视觉大语言模型(VisionLLM-based)的多模态融合网络用于声门癌检测,称为MMGC-Net。通过整合图像和文本模态,多模态模型可以捕获互补信息,从而实现更准确和稳健的预测。在本文中,我们从中山大学附属第一医院收集了一个名为SYSU1H的私有真实声门癌数据集,包含5,799个图文对。我们利用图像编码器和额外的Q-Former来提取视觉嵌入,并利用Large Language Model Meta AI (Llama3)来获取文本嵌入。然后通过喉部特征融合块整合这些模态,实现图像和文本特征的全面整合,从而提高声门癌识别性能。在SYSU1H数据集上的大量实验表明,MMGC-Net能够达到最先进的性能,优于以往的多模态模型。
英文摘要:
The early detection of glottic carcinoma is critical for improving patient outcomes, as it enables timely intervention, preserves vocal function, and significantly reduces the risk of tumor progression and metastasis. However, the similarity in morphology between glottic carcinoma and vocal cord dysplasia results in suboptimal detection accuracy. To address this issue, we propose a vision large language model-based (VisionLLM-based) multimodal fusion network for glottic carcinoma detection, known as MMGC-Net. By integrating image and text modalities, multimodal models can capture complementary information, leading to more accurate and robust predictions. In this paper, we collect a private real glottic carcinoma dataset named SYSU1H from the First Affiliated Hospital of Sun Yat-sen University, with 5,799 image-text pairs. We leverage an image encoder and additional Q-Former to extract vision embeddings and the Large Language Model Meta AI (Llama3) to obtain text embeddings. These modalities are then integrated through a laryngeal feature fusion block, enabling a comprehensive integration of image and text features, thereby improving the glottic carcinoma identification performance. Extensive experiments on the SYSU1H dataset demonstrate that MMGC-Net can achieve state-of-the-art performance, which is superior to previous multimodal models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。