arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.11558cs.CVcs.CL
收藏

利用场景知识推进视觉定位:基准与方法

Advancing Visual Grounding with Scene Knowledge: Benchmark and Method

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Sun Yat-sen University(中山大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Tencent AI Lab(腾讯人工智能实验室);AI 3 Institute, Fudan University(复旦大学人工智能创新与产业研究院)

Zhihong Chen, Ruifei Zhang, Yibing Song, Xiang Wan, Guanbin Li

AI总结:

针对现有视觉定位数据集推理要求低的问题,本文提出场景知识引导的SK-VG基准及两种三输入处理方法,验证了方法有效性并指出仍存性能与可解释性提升空间。

摘要:

视觉定位(VG)旨在建立视觉与语言之间的细粒度对齐。理想情况下,它可作为视觉-语言模型的测试平台,用于评估模型对图像和文本的理解能力,以及在二者联合空间中的推理能力。然而,现有大多数VG数据集均采用简单描述文本构建,无需对图像和文本进行充分推理。近期一项研究已证实这一点:一个未经预训练的、基于LSTM的简单文本编码器,就能在主流VG数据集上取得最优性能。因此,本文提出了全新的场景知识引导视觉定位基准(SK-VG),该基准中仅靠图像内容和指代表达不足以定位目标物体,迫使模型具备对长文本场景知识的推理能力。为完成该任务,我们提出两种可接受三类型输入的方法:前者在图像-查询交互前将知识嵌入图像特征;后者利用语言结构辅助计算图文匹配度。我们开展了大量实验分析上述方法,结果表明所提方法取得了可观效果,但在性能和可解释性方面仍有提升空间。数据集和代码可在https://github.com/zhjohnchan/SK-VG 获取。

英文摘要:

Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Ideally, it can be a testbed for vision-and-language models to evaluate their understanding of the images and texts and their reasoning abilities over their joint space. However, most existing VG datasets are constructed using simple description texts, which do not require sufficient reasoning over the images and texts. This has been demonstrated in a recent study~\cite{luo2022goes}, where a simple LSTM-based text encoder without pretraining can achieve state-of-the-art performance on mainstream VG datasets. Therefore, in this paper, we propose a novel benchmark of \underline{S}cene \underline{K}nowledge-guided \underline{V}isual \underline{G}rounding (SK-VG), where the image content and referring expressions are not sufficient to ground the target objects, forcing the models to have a reasoning ability on the long-form scene knowledge. To perform this task, we propose two approaches to accept the triple-type input, where the former embeds knowledge into the image features before the image-query interaction; the latter leverages linguistic structure to assist in computing the image-text matching. We conduct extensive experiments to analyze the above methods and show that the proposed approaches achieve promising results but still leave room for improvement, including performance and interpretability. The dataset and code are available at \url{https://github.com/zhjohnchan/SK-VG}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑