一种结合不同模态侧信息的主题感知摘要框架
A Topic-aware Summarization Framework with Different Modal Side Information
AI总结：
提出一种通用主题感知摘要框架，通过统一主题编码器和三元组对比学习机制灵活融合多模态侧信息，在三个基准数据集上显著超越强基线。
摘要：
自动摘要在网络文档呈指数级增长中扮演着重要角色。在CNN.com和WikiHow.com等内容网站上，主文档通常伴随各种侧信息以吸引注意力并便于理解，例如视频、图像和查询。这些信息可用于实现更好的摘要，因为它们通常显式或隐式地提及文章的本质。然而，现有的大多数侧信息感知摘要方法被设计为仅结合单模态或多模态侧信息，无法有效相互适应。在本文中，我们提出了一个通用摘要框架，可以灵活结合各种模态的侧信息。设计带有侧信息的灵活摘要模型的主要挑战包括：(1) 侧信息可以是文本或视觉格式，模型需要将其与文档对齐并统一到相同的语义空间中；(2) 侧输入可能包含来自不同方面的信息，模型应识别对摘要有用的方面。为了解决这两个挑战，我们首先提出一个统一主题编码器，它联合从文档和各种侧信息中发现潜在主题。学习到的主题通过主题感知交互，在图编码器中灵活地桥接并引导多个输入之间的信息流。其次，我们提出一种三元组对比学习机制，将单模态或多模态信息对齐到统一的语义空间中，通过更好地理解文档和侧信息来提升摘要质量。结果表明，我们的模型在三个公开的单模态或多模态基准摘要数据集上显著超越了强基线。
英文摘要：
Automatic summarization plays an important role in the exponential document growth on the Web. On content websites such as CNN.com and WikiHow.com, there often exist various kinds of side information along with the main document for attention attraction and easier understanding, such as videos, images, and queries. Such information can be used for better summarization, as they often explicitly or implicitly mention the essence of the article. However, most of the existing side-aware summarization methods are designed to incorporate either single-modal or multi-modal side information, and cannot effectively adapt to each other. In this paper, we propose a general summarization framework, which can flexibly incorporate various modalities of side information. The main challenges in designing a flexible summarization model with side information include: (1) the side information can be in textual or visual format, and the model needs to align and unify it with the document into the same semantic space, (2) the side inputs can contain information from various aspects, and the model should recognize the aspects useful for summarization. To address these two challenges, we first propose a unified topic encoder, which jointly discovers latent topics from the document and various kinds of side information. The learned topics flexibly bridge and guide the information flow between multiple inputs in a graph encoder through a topic-aware interaction. We secondly propose a triplet contrastive learning mechanism to align the single-modal or multi-modal information into a unified semantic space, where the summary quality is enhanced by better understanding the document and side information. Results show that our model significantly surpasses strong baselines on three public single-modal or multi-modal benchmark summarization datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
