arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.03751cs.CV
收藏

采用HINT:利用简化使视觉和语言模型更加基础

Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded

Georgia Institute of Technology(佐治亚理工学院);Oregon State University(俄勒冈州立大学);Samsung Research(三星研究院);Facebook AI Research(Facebook人工智能研究院)

Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, Devi Parikh

摘要:

许多视觉和语言模型都有着糟糕的视觉基础--通常依赖于容易学习的语言先验知识,而不是基于图像中的视觉概念做出决定。在这项工作中,我们提出了一种通用的方法,称为人类重要性感知网络调整(HINT),有效地利用人类的演示,以提高视觉接地。HINT鼓励深度网络对与人类相同的输入区域敏感。我们的方法优化了人类注意力地图和基于梯度的网络重要性之间的一致性-确保模型学习不仅仅是看,而是依赖于人类在进行预测时发现与任务相关的视觉概念。我们将HINT应用于视觉问答和图像描述任务,在使用人类注意力演示仅6%的训练数据的情况下,在惩罚过度依赖语言先验(VQA-CP和鲁棒字幕)的分裂上优于顶级方法。

英文摘要:

Many vision and language models suffer from poor visual grounding - often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the image. In this work, we propose a generic approach called Human Importance-aware Network Tuning (HINT) that effectively leverages human demonstrations to improve visual grounding. HINT encourages deep networks to be sensitive to the same input regions as humans. Our approach optimizes the alignment between human attention maps and gradient-based network importances - ensuring that models learn not just to look at but rather rely on visual concepts that humans found relevant for a task when making predictions. We apply HINT to Visual Question Answering and Image Captioning tasks, outperforming top approaches on splits that penalize over-reliance on language priors (VQA-CP and robust captioning) using human attention demonstrations for just 6% of the training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑