通过使用共同注意力对齐符号和对象来理解视觉广告
Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
摘要：
我们解决的问题是理解视觉广告，在给定的广告图像，我们的目标是排名适当的人类生成的语句描述的广告的目的。这个问题通常通过联合嵌入图像和候选语句来建立对应关系来解决。解码视觉广告需要推断图像中引用的语义和符号细微差别，并且现有方法可能无法捕获这种关联，尤其是对于弱注释符号。为了创建更好的嵌入，我们利用注意力机制将图像提案与符号相关联，从而有效地从对齐的多模态表示中聚合信息。我们提出了一种多跳协同注意力机制，迭代地细化注意力图，以确保准确的注意力估计。我们的基于注意力的嵌入模型是由最大保证金损失函数指导的端到端学习的。我们表明，我们的模型在基准广告数据集上的表现优于其他基线，并且还显示了定性结果，以突出使用多跳共同关注的优势。
英文摘要：
We tackle the problem of understanding visual ads where given an ad image, our goal is to rank appropriate human generated statements describing the purpose of the ad. This problem is generally addressed by jointly embedding images and candidate statements to establish correspondence. Decoding a visual ad requires inference of both semantic and symbolic nuances referenced in an image and prior methods may fail to capture such associations especially with weakly annotated symbols. In order to create better embeddings, we leverage an attention mechanism to associate image proposals with symbols and thus effectively aggregate information from aligned multimodal representations. We propose a multihop co-attention mechanism that iteratively refines the attention map to ensure accurate attention estimation. Our attention based embedding model is learned end-to-end guided by a max-margin loss function. We show that our model outperforms other baselines on the benchmark Ad dataset and also show qualitative results to highlight the advantages of using multihop co-attention.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
