两只眼睛，两种视角，最终一个总结！迈向多模态多任务知识注入的医疗对话摘要生成
Two eyes, Two views, and finally, One summary! Towards Multi-modal Multi-tasking Knowledge-Infused Medical Dialogue Summarization
AI总结：
提出MMK-Summation模型，通过多模态多任务知识注入同时生成医疗关注、医生印象及整体摘要，超越多个基线，验证了知识引导的多任务多模态医疗对话摘要的有效性。
摘要：
我们通常分两个阶段总结多方对话：先按同质单元进行分块，再对各块进行摘要。因此，我们假设同质说话者分块与整体摘要任务之间存在相关性。在本工作中，我们研究了一种多面方法的有效性，该方法同时生成医疗关注点、医生印象和整体视图的摘要。我们引入了一种多模态、多任务、知识注入的医疗对话摘要生成（MMK-Summation）模型，该模型通过门控机制结合基于适配器的微调，实现多模态信息整合。MMK-Summation模型以对话为输入，基于上下文提取相关外部知识，将知识及对话中的视觉线索整合到文本内容中，最终生成涵盖医疗关注点、医生印象和全面概述的简洁摘要。该模型在所有评估指标（包括人工评估）上均超越了多个基线和传统摘要模型，这有力地证明了知识引导的多任务、多模态医疗对话摘要的有效性。代码可在 https://github.com/NLP-RL/MMK-Summation 获取。
英文摘要：
We often summarize a multi-party conversation in two stages: chunking with homogeneous units and summarizing the chunks. Thus, we hypothesize that there exists a correlation between homogeneous speaker chunking and overall summarization tasks. In this work, we investigate the effectiveness of a multi-faceted approach that simultaneously produces summaries of medical concerns, doctor impressions, and an overall view. We introduce a multi-modal, multi-tasking, knowledge-infused medical dialogue summary generation (MMK-Summation) model, which is incorporated with adapter-based fine-tuning through a gated mechanism for multi-modal information integration. The model, MMK-Summation, takes dialogues as input, extracts pertinent external knowledge based on the context, integrates the knowledge and visual cues from the dialogues into the textual content, and ultimately generates concise summaries encompassing medical concerns, doctor impressions, and a comprehensive overview. The introduced model surpasses multiple baselines and traditional summarization models across all evaluation metrics (including human evaluation), which firmly demonstrates the efficacy of the knowledge-guided multi-tasking, multimodal medical conversation summarization. The code is available at https://github.com/NLP-RL/MMK-Summation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
