基于图文联合语料库的视觉主题自动发现
Automatic Visual Theme Discovery from Joint Image and Text Corpora
摘要:
一种流行的语义图像理解方法是手动标记图像与关键字,然后学习从视觉特征到关键字的映射。人工标记图像是一个主观的过程,相同或非常相似的视觉内容通常使用不同的关键字标记。此外,并非所有的标签都对视觉内容具有相同的描述能力,并且来自自然语言的大量词汇可能导致非常多样化的关键字集合。在本文中,我们提出了一个无监督的视觉主题发现框架作为一个更好的(更紧凑,高效和有效的)替代语义表示的视觉内容。我们首先表明,基于标签的注释缺乏一致性和紧凑性描述视觉上相似的内容。然后,我们根据包含标签的图像的视觉特征来学习标签之间的视觉相似性。同时,我们使用一种自然语言处理技术(词嵌入)来度量标签之间的语义相似度。最后,我们使用谱聚类算法根据标签的视觉相似性和语义相似性度量将其聚类到视觉主题中。我们通过用户研究来评估我们的非监督算法发现的视觉主题的有效性和合理性,并取得了令人满意的结果。然后,我们设计了三个常见的计算机视觉任务,基于示例的图像搜索,基于关键字的图像搜索和图像标记,以探索我们的视觉主题发现框架的潜在应用。在实验中,视觉主题显著优于标签的语义图像理解,并实现了国家的艺术表现在所有三个任务。这再次证明了所提出的框架的有效性和通用性。
英文摘要:
A popular approach to semantic image understanding is to manually tag images with keywords and then learn a mapping from vi- sual features to keywords. Manually tagging images is a subjective pro- cess and the same or very similar visual contents are often tagged with different keywords. Furthermore, not all tags have the same descriptive power for visual contents and large vocabulary available from natural language could result in a very diverse set of keywords. In this paper, we propose an unsupervised visual theme discovery framework as a better (more compact, efficient and effective) alternative to semantic represen- tation of visual contents. We first show that tag based annotation lacks consistency and compactness for describing visually similar contents. We then learn the visual similarity between tags based on the visual features of the images containing the tags. At the same time, we use a natural language processing technique (word embedding) to measure the seman- tic similarity between tags. Finally, we cluster tags into visual themes based on their visual similarity and semantic similarity measures using a spectral clustering algorithm. We conduct user studies to evaluate the effectiveness and rationality of the visual themes discovered by our unsu- pervised algorithm and obtains promising result. We then design three common computer vision tasks, example based image search, keyword based image search and image labelling to explore potential applica- tion of our visual themes discovery framework. In experiments, visual themes significantly outperforms tags on semantic image understand- ing and achieve state-of-art performance in all three tasks. This again demonstrate the effectiveness and versatility of proposed framework.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。