arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.04971cs.CLcs.CV
收藏

理解生成场景描述的V&L模型中的跨模态交互

Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions

University of Malta(马耳他大学);Universiteit Utrecht(乌得勒支大学)

Michele Cafagna, Kees van Deemter, Albert Gatt

摘要:

图像描述模型倾向于以对象为中心的方式描述图像,强调可见对象。但图像描述也可以从对象中抽象出来,描述所描绘的场景类型。在本文中,我们探讨了一个国家的最先进的视觉和语言模型,VinVL,字幕图像在场景级别使用(1)一个新的数据集,对图像与对象为中心的场景描述。通过(2)对微调效果的深入分析,我们表明(3)少量策划数据足以生成场景描述,而不会失去识别场景中对象级别概念的能力;与生成以对象为中心的描述时相比,该模型获得了更全面的图像视图。我们讨论了这些结果和见解之间的相似之处,从计算和认知科学研究的场景感知。

英文摘要:

Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potential of a state-of-the-art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions. Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when object-centric descriptions are generated. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑