CLIPSyntel：CLIP与LLM协同用于医疗领域多模态问题摘要生成
CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare
AI总结：
针对现有医疗问题摘要仅依赖文本的不足，本文提出MMQS多模态数据集及CLIP与LLM协同的四模块框架，验证了视觉信息可提升医疗摘要质量，为个性化医疗研究奠基。
摘要：
在现代医疗时代，快速生成医疗问题摘要对于知情且及时的患者护理至关重要。尽管医疗数据的复杂性和体量不断增加，现有研究仅聚焦于基于文本的摘要生成，忽略了视觉信息的整合。我们意识到将文本查询与病症视觉表征相结合的未开发潜力，因此提出了多模态医疗问题摘要（Multimodal Medical Question Summarization，MMQS）数据集。该数据集是本研究的一项核心贡献，将医疗查询与视觉辅助资料配对，助力更丰富、更细致地理解患者需求。我们还提出了一个利用对比语言-图像预训练（Contrastive Language Image Pretraining，CLIP）和大语言模型（Large Language Models，LLMs）能力的框架，包含四个模块：识别病症、生成相关上下文、过滤医疗概念以及生成具备视觉感知的摘要。我们的综合框架借助多模态基础模型CLIP和各类通用LLM的能力，包含四大核心模块：病症识别模块、相关上下文生成模块、用于提炼相关医疗概念与知识的上下文过滤模块，以及最终用于生成具备视觉感知的医疗问题摘要的通用LLM。借助我们的MMQS数据集，我们展示了图像中的视觉线索如何提升医疗层面细致入微的摘要生成效果。这种多模态方法不仅优化了医疗领域的决策流程，还能促进对患者查询的更细致理解，为个性化响应式医疗护理的未来研究奠定基础。
英文摘要：
In the era of modern healthcare, swiftly generating medical question summaries is crucial for informed and timely patient care. Despite the increasing complexity and volume of medical data, existing studies have focused solely on text-based summarization, neglecting the integration of visual information. Recognizing the untapped potential of combining textual queries with visual representations of medical conditions, we introduce the Multimodal Medical Question Summarization (MMQS) Dataset. This dataset, a major contribution to our work, pairs medical queries with visual aids, facilitating a richer and more nuanced understanding of patient needs. We also propose a framework, utilizing the power of Contrastive Language Image Pretraining(CLIP) and Large Language Models(LLMs), consisting of four modules that identify medical disorders, generate relevant context, filter medical concepts, and craft visually aware summaries. Our comprehensive framework harnesses the power of CLIP, a multimodal foundation model, and various general-purpose LLMs, comprising four main modules: the medical disorder identification module, the relevant context generation module, the context filtration module for distilling relevant medical concepts and knowledge, and finally, a general-purpose LLM to generate visually aware medical question summaries. Leveraging our MMQS dataset, we showcase how visual cues from images enhance the generation of medically nuanced summaries. This multimodal approach not only enhances the decision-making process in healthcare but also fosters a more nuanced understanding of patient queries, laying the groundwork for future research in personalized and responsive medical care
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
