考虑Wasserstein图匹配的半监督图像描述
Semi-Supervised Image Captioning Considering Wasserstein Graph Matching
AI总结：
提出考虑Wasserstein图匹配的半监督图像描述方法SSIC-WGM，利用场景图作为中间信息实现模态间和模态内一致性约束，从而高效利用未描述图像学习更合理的视觉到语言映射函数。
摘要：
图像描述能够自动为给定图像生成描述，其关键挑战在于学习从视觉特征到自然语言特征的映射函数。现有方法多为监督方法，即训练集中每张图像都有对应的句子。然而，考虑到描述图像总是需要大量人力，在实际应用中，我们通常拥有数量有限的已描述图像（即图文对）和大量未描述图像。因此，面临的一个困境便是“半监督图像描述”。为解决此问题，我们提出了一种考虑Wasserstein图匹配的新型半监督图像描述方法（SSIC-WGM），转而采用原始图像输入来监督生成的句子。与传统的单模态半监督方法不同，半监督跨模态学习的难点在于构建异构模态之间可比较的中间信息。本文中，SSIC-WGM采用成功的场景图作为中间信息，并从两方面约束生成的句子：1）模态间一致性。SSIC-WGM分别构建原始图像和生成句子的场景图，然后采用Wasserstein距离来更好地度量不同图的区域嵌入之间的相似度。2）模态内一致性。SSIC-WGM对原始图像采用数据增强技术，随后约束增强图像与生成句子之间的一致性。因此，SSIC-WGM结合跨模态伪监督和结构不变性度量，以高效利用未描述图像，并学习更合理的映射函数。
英文摘要：
Image captioning can automatically generate captions for the given images, and the key challenge is to learn a mapping function from visual features to natural language features. Existing approaches are mostly supervised ones, i.e., each image has a corresponding sentence in the training set. However, considering that describing images always requires a huge of manpower, we usually have limited amount of described images (i.e., image-text pairs) and a large number of undescribed images in real-world applications. Thereby, a dilemma is the "Semi-Supervised Image Captioning". To solve this problem, we propose a novel Semi-Supervised Image Captioning method considering Wasserstein Graph Matching (SSIC-WGM), which turns to adopt the raw image inputs to supervise the generated sentences. Different from traditional single modal semi-supervised methods, the difficulty of semi-supervised cross-modal learning lies in constructing intermediately comparable information among heterogeneous modalities. In this paper, SSIC-WGM adopts the successful scene graphs as intermediate information, and constrains the generated sentences from two aspects: 1) inter-modal consistency. SSIC-WGM constructs the scene graphs of the raw image and generated sentence respectively, then employs the wasserstein distance to better measure the similarity between region embeddings of different graphs. 2) intra-modal consistency. SSIC-WGM takes the data augmentation techniques for the raw images, then constrains the consistency among augmented images and generated sentences. Consequently, SSIC-WGM combines the cross-modal pseudo supervision and structure invariant measure for efficiently using the undescribed images, and learns more reasonable mapping function.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
