SITransformer：面向极端多模态摘要的共享信息引导Transformer
SITransformer: Shared Information-Guided Transformer for Extreme Multimodal Summarization
AI总结：
针对极端多模态摘要（XMSMO）中多模态数据含无关信息易误导模型的问题，提出SITransformer，通过共享信息提取器和跨模态交互模块提升视频与文本摘要质量，代码将公开。
摘要：
带多模态输出的极端多模态摘要（XMSMO）通过整合各类信息为单模态生成极度简洁却丰富的摘要，成为一种颇具吸引力的摘要方法。现有方法忽略了多模态数据常包含更多与主题无关的信息这一问题，这可能误导模型生成不准确的摘要，对于极短摘要尤其如此。本文提出SITransformer，一种面向极端多模态摘要的共享信息引导Transformer。它具有共享信息引导流水线，包含跨模态共享信息提取器和跨模态交互模块。提取器通过设计包含可微分top-k选择器和共享信息引导门控单元的新型过滤过程，从不同模态中提炼语义共享的显著信息，从而识别跨模态的共同、显著且相关的内容。接着，开发带有跨模态注意力的Transformer，在共享信息引导下进行模态内和模态间学习以生成极端摘要。综合实验表明，SITransformer显著提升了XMSMO中视频和文本摘要的质量。我们的代码将在https://github.com/SichengLeoLiu/MMAsia24-XMSMO公开。
英文摘要：
Extreme Multimodal Summarization with Multimodal Output (XMSMO) becomes an attractive summarization approach by integrating various types of information to create extremely concise yet informative summaries for individual modalities. Existing methods overlook the issue that multimodal data often contains more topic irrelevant information, which can mislead the model into producing inaccurate summaries especially for extremely short ones. In this paper, we propose SITransformer, a Shared Information-guided Transformer for extreme multimodal summarization. It has a shared information guided pipeline which involves a cross-modal shared information extractor and a cross-modal interaction module. The extractor formulates semantically shared salient information from different modalities by devising a novel filtering process consisting of a differentiable top-k selector and a shared-information guided gating unit. As a result, the common, salient, and relevant contents across modalities are identified. Next, a transformer with cross-modal attentions is developed for intra- and inter-modality learning with the shared information guidance to produce the extreme summary. Comprehensive experiments demonstrate that SITransformer significantly enhances the summarization quality for both video and text summaries for XMSMO. Our code will be publicly available at https://github.com/SichengLeoLiu/MMAsia24-XMSMO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
