arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1707.00683cs.CVcs.CLcs.LG
收藏

通过语言调节早期视觉加工

Modulating early visual processing by language

University of Montreal(蒙特利尔大学);Univ. Lille(里尔大学);CNRS(法国国家科学研究中心);Centrale Lille(里尔中央理工学院);Inria(法国国家信息与自动化研究所);UMR 9189 CRIStAL(法国国家科学研究中心计算机、信号与自动化研究所);Criteo;Google Brain(谷歌大脑);DeepMind;CIFAR(加拿大高等研究院)

Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, Aaron Courville

摘要:

人们通常认为,语言是指高层次的视觉概念,而离开低层次的视觉处理不受影响。这一观点在语言-视觉任务的计算模型中占据主导地位,其中视觉和语言输入在被融合成单个表示之前大多是独立处理的。在本文中,我们偏离了这个经典的管道,并建议调制的\emph{entire visual processing}的语言输入。具体来说,我们将预训练残差网络(ResNet)的批量归一化参数设置在语言嵌入上。这种方法,我们称之为调制RESnet(\MRN),显着提高了两个视觉问答任务的强基线。我们的消融研究表明,从视觉处理的早期阶段进行调节是有益的。

英文摘要:

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (\MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑