arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1904.08709cs.IRcs.CLcs.CV
收藏

知识丰富的图像主旨理解超越字面意义

Knowledge-rich Image Gist Understanding Beyond Literal Meaning

University of Mannheim(曼海姆大学);University of New Hampshire(新罕布什尔大学)

Lydia Weiland, Ioana Hulpus, Simone Paolo Ponzetto, Wolfgang Effelsberg, Laura Dietz

摘要:

我们调查的问题,理解的信息(要旨)所传达的图像和他们的标题,发现,例如,在网站或新闻文章。为此,我们提出了一种方法来捕获的图像标题对的意义的基础上,大量的机器可读的知识,以前已被证明是非常有效的文本理解。我们的方法识别对象的内涵超出其外延:其中大多数图像理解方法都集中在对象的外延上,即,它们的字面意义,我们的工作解决了内涵的识别,即,物体的标志性意义,理解图像的信息。我们认为图像理解的任务,表示一个图像的标题对的基础上,广泛覆盖的词汇表的概念,如维基百科提供的一个,和铸造要点检测作为一个概念排名问题与图像标题对查询。为了能够彻底调查要点理解的问题,我们制作了一个黄金标准,包括300多个图像-标题对和8,000多个要点注释,涵盖了不同抽象层次的各种主题。我们使用这个数据集来实验性地基准测试来自异构源(即图像和文本)的信号的贡献。平均精度(MAP)为0.69的最佳结果表明,通过结合两个维度,我们能够比单独使用语言或视觉信息时更好地理解图像-标题对的含义。我们测试了我们的要点检测方法在接收自动生成的输入时的鲁棒性,即,使用自动生成的图像标签或生成的字幕,并证明端到端自动化过程的可行性。

英文摘要:

We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that has previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their literal meaning, our work addresses the identification of connotations, i.e., iconic meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. To enable a thorough investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8,000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the meaning of our image-caption pairs than when using language or vision information alone. We test the robustness of our gist detection approach when receiving automatically generated input, i.e., using automatically generated image tags or generated captions, and prove the feasibility of an end-to-end automated process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑