VQS：在VQA中将分割与问题和答案联系起来，以获得监督注意力，以及关注语义的语义分割
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
摘要：
丰富而密集的人类标记数据集是最近视觉-语言理解进步的主要推动因素之一。许多看似遥远的注释（例如，语义分割和视觉问答（VQA））是内在联系的，因为它们揭示了人类对相同视觉场景的理解的不同层次和视角--甚至是同一组图像（例如，COCO）。COCO的流行与这些注释和任务相关。明确地将它们联系起来可能会对单个任务以及统一的视觉和语言建模都有很大的好处。我们提出了初步的工作，链接的实例分割提供的COCO的问题和答案（QA）的VQA数据集，并命名收集的链接视觉问题和分割答案（VQS）。它们在以前独立的任务之间转移了人类的监督，为现有问题提供了更有效的杠杆，并为新的研究问题和模型打开了大门。本文研究了VQS数据的两个应用：VQA的监督注意和一种新的以问题为中心的语义分割任务。对于前者，我们通过简单地增加多层感知器的一些注意力特征，这些注意力特征是使用分割QA链接作为显式监督来学习的，从而在VQA真实多项选择任务上获得了最先进的结果。为了把后者的角度来看，我们研究了两种似是而非的方法，并将它们与Oracle方法进行比较，假设在测试阶段给出了实例分割。
英文摘要：
Rich and dense human labeled datasets are among the main enabling factors for the recent advance on vision-language understanding. Many seemingly distant annotations (e.g., semantic segmentation and visual question answering (VQA)) are inherently connected in that they reveal different levels and perspectives of human understandings about the same visual scenes --- and even the same set of images (e.g., of COCO). The popularity of COCO correlates those annotations and tasks. Explicitly linking them up may significantly benefit both individual tasks and the unified vision and language modeling. We present the preliminary work of linking the instance segmentations provided by COCO to the questions and answers (QAs) in the VQA dataset, and name the collected links visual questions and segmentation answers (VQS). They transfer human supervision between the previously separate tasks, offer more effective leverage to existing problems, and also open the door for new research problems and models. We study two applications of the VQS data in this paper: supervised attention for VQA and a novel question-focused semantic segmentation task. For the former, we obtain state-of-the-art results on the VQA real multiple-choice task by simply augmenting the multilayer perceptrons with some attention features that are learned using the segmentation-QA links as explicit supervision. To put the latter in perspective, we study two plausible methods and compare them to an oracle method assuming that the instance segmentations are given at the test stage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
