CISum：学习跨模态交互以增强多模态语义覆盖以实现多模态摘要
CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization
摘要：
多模态摘要（MS）旨在从多模态输入生成摘要。以前的工作主要集中在文本语义覆盖率指标，如ROUGE，认为视觉内容作为补充数据。因此，该摘要不能有效地涵盖不同模态的语义。本文提出了一个多任务跨模态学习框架（CISum），通过学习多模态文章中的跨模态交互来提高多模态语义覆盖率。为了获得视觉语义，我们将图像转换为视觉描述的基础上与文本内容的相关性。然后，视觉描述和文本内容被融合以生成文本摘要来捕获多模态内容的语义，并且最相关的图像被选择作为视觉摘要。此外，我们设计了一个自动的多模态语义覆盖度量来评估性能。实验结果表明，CISum在保持ROUGE和BLEU优秀性能的同时，在多模态语义覆盖度量方面优于基线。
英文摘要：
Multimodal summarization (MS) aims to generate a summary from multimodal input. Previous works mainly focus on textual semantic coverage metrics such as ROUGE, which considers the visual content as supplemental data. Therefore, the summary is ineffective to cover the semantics of different modalities. This paper proposes a multi-task cross-modality learning framework (CISum) to improve multimodal semantic coverage by learning the cross-modality interaction in the multimodal article. To obtain the visual semantics, we translate images into visual descriptions based on the correlation with text content. Then, the visual description and text content are fused to generate the textual summary to capture the semantics of the multimodal content, and the most relevant image is selected as the visual summary. Furthermore, we design an automatic multimodal semantics coverage metric to evaluate the performance. Experimental results show that CISum outperforms baselines in multimodal semantics coverage metrics while maintaining the excellent performance of ROUGE and BLEU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
