LOIS：面向视觉问答的实例语义外探
LOIS: Looking Out of Instance Semantics for Visual Question Answering
AI总结：
针对VQA中边界框限制语义理解的问题，提出无边界框的LOIS框架，通过细粒度特征描述与模态内/模态间关系注意力模块建模视觉语义关系，在四个基准数据集上有效提升了视觉推理能力。
摘要：
视觉问答（VQA）作为一个多模态任务，一直被广泛研究，它需要努力弥合视觉与语言以正确推断答案。最近的尝试开发了各种基于注意力的模块来解决VQA任务。然而，模型推理的性能在很大程度上受到用于语义理解的视觉处理的瓶颈限制。大多数现有的检测方法依赖于边界框，这使得VQA模型在理解图像中物体语义的因果关系并正确推断上下文信息方面仍面临严峻挑战。为此，在本研究中，我们提出了一种无需边界框的更精细的模型框架，称为Looking Out of Instance Semantics（LOIS），以解决这一重要问题。LOIS能够产生更细粒度的特征描述来生成视觉事实。此外，为了克服由实例掩码引起的标签模糊问题，设计了两种关系注意力模块：1）模态内和2）模态间，用于从不同的多视图特征中推断正确答案。具体而言，我们实现了一个相互关系注意力模块，以建模实例对象与背景信息之间复杂且更深层的视觉语义关系。此外，我们提出的注意力模型可以通过聚焦于重要的与单词相关的问题来进一步分析显著的图像区域。在四个基准VQA数据集上的实验结果证明，我们提出的方法在提高视觉推理能力方面具有良好的性能。
英文摘要：
Visual question answering (VQA) has been intensively studied as a multimodal task that requires effort in bridging vision and language to infer answers correctly. Recent attempts have developed various attention-based modules for solving VQA tasks. However, the performance of model inference is largely bottlenecked by visual processing for semantics understanding. Most existing detection methods rely on bounding boxes, remaining a serious challenge for VQA models to understand the causal nexus of object semantics in images and correctly infer contextual information. To this end, we propose a finer model framework without bounding boxes in this work, termed Looking Out of Instance Semantics (LOIS) to tackle this important issue. LOIS enables more fine-grained feature descriptions to produce visual facts. Furthermore, to overcome the label ambiguity caused by instance masks, two types of relation attention modules: 1) intra-modality and 2) inter-modality, are devised to infer the correct answers from the different multi-view features. Specifically, we implement a mutual relation attention module to model sophisticated and deeper visual semantic relations between instance objects and background information. In addition, our proposed attention model can further analyze salient image regions by focusing on important word-related questions. Experimental results on four benchmark VQA datasets prove that our proposed method has favorable performance in improving visual reasoning capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
