AlignVE:基于对齐关系的视觉蕴涵识别
AlignVE: Visual Entailment Recognition Based on Alignment Relations
摘要:
视觉蕴涵(Visual entailment,VE)是近年来出现的视觉和语言理解任务中的一个特殊任务,它是识别一个假设文本的语义是否可以从给定的前提图像中推断出来。目前,大多数现有的VE方法是从视觉问答的方法。他们通过量化多模态内容语义特征中假设和前提之间的相似性来识别视觉蕴涵。然而,这种方法忽略了价值工程的独特性质的前提和假设之间的关系推理。因此,在本文中,一个新的架构称为AlignVE提出了解决视觉蕴涵问题的关系交互方法。它将前提和假设之间的关系建模为对齐矩阵。然后引入一个池化操作来得到一个固定大小的特征向量。最后经过全连接层和归一化层完成分类。实验结果表明,在SNLI-VE数据集上,该模型的准确率达到了72.45%,在相同的设置下,优于以往的基于内容的模型.
英文摘要:
Visual entailment (VE) is to recognize whether the semantics of a hypothesis text can be inferred from the given premise image, which is one special task among recent emerged vision and language understanding tasks. Currently, most of the existing VE approaches are derived from the methods of visual question answering. They recognize visual entailment by quantifying the similarity between the hypothesis and premise in the content semantic features from multi modalities. Such approaches, however, ignore the VE's unique nature of relation inference between the premise and hypothesis. Therefore, in this paper, a new architecture called AlignVE is proposed to solve the visual entailment problem with a relation interaction method. It models the relation between the premise and hypothesis as an alignment matrix. Then it introduces a pooling operation to get feature vectors with a fixed size. Finally, it goes through the fully-connected layer and normalization layer to complete the classification. Experiments show that our alignment-based architecture reaches 72.45\% accuracy on SNLI-VE dataset, outperforming previous content-based models under the same settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。