arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.01982cs.CVcs.AIcs.CL
收藏

你的图像是一个好的故事讲述者吗?

Is Your Image a Good Storyteller?

Shanghai Jiao Tong University(上海交通大学);China Merchants Bank Credit Card Center(招商银行信用卡中心);University of Texas at Arlington(德克萨斯大学阿灵顿分校)

Xiujie Song, Xiaoyi Pang, Haifeng Tang, Mengyue Wu, Kenny Q. Zhu

AI总结:

本文提出图像语义评估(ISA)任务,通过语言方法自动评估图像语义复杂度,并构建首个ISA数据集,以支持挖掘和生成语义丰富的图像,助力人类认知评估与AI应用。

摘要:

在实体层面量化图像复杂度是直接的,但语义复杂度的评估在很大程度上被忽视了。事实上,不同图像之间的语义复杂度存在差异。语义更丰富的图像能够讲述生动且引人入胜的故事,并提供广泛的应用场景。例如,Cookie Theft 图片就是这样的图像,由于其较高的语义复杂度,被广泛用于评估人类语言和认知能力。此外,语义丰富的图像有助于视觉模型的发展,因为语义有限的图像对它们来说正变得不那么具有挑战性。然而,这类图像是稀缺的,这凸显了对更多此类图像的需求。例如,需要更多像 Cookie Theft 这样的图像,以满足来自不同文化背景和时代的人们的需求。评估语义复杂度需要人类专家和经验证据。自动评估图像的语义丰富程度将是挖掘或生成更多具有丰富语义的图像的第一步,并有益于人类认知评估、人工智能及其他各种应用。为此,我们提出了图像语义评估(ISA)任务来解决这个问题。我们引入了第一个 ISA 数据集,以及一种利用语言来解决这一视觉问题的新方法。在我们数据集上的实验证明了我们方法的有效性。

英文摘要:

Quantifying image complexity at the entity level is straightforward, but the assessment of semantic complexity has been largely overlooked. In fact, there are differences in semantic complexity across images. Images with richer semantics can tell vivid and engaging stories and offer a wide range of application scenarios. For example, the Cookie Theft picture is such a kind of image and is widely used to assess human language and cognitive abilities due to its higher semantic complexity. Additionally, semantically rich images can benefit the development of vision models, as images with limited semantics are becoming less challenging for them. However, such images are scarce, highlighting the need for a greater number of them. For instance, there is a need for more images like Cookie Theft to cater to people from different cultural backgrounds and eras. Assessing semantic complexity requires human experts and empirical evidence. Automatic evaluation of how semantically rich an image will be the first step of mining or generating more images with rich semantics, and benefit human cognitive assessment, Artificial Intelligence, and various other applications. In response, we propose the Image Semantic Assessment (ISA) task to address this problem. We introduce the first ISA dataset and a novel method that leverages language to solve this vision problem. Experiments on our dataset demonstrate the effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑