arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.14352cs.CV
收藏

在另一面寻找它:一个视点适应的匹配编码器,用于更改字幕

Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning

Nanyang Technological University(南洋理工大学);Adobe Research(Adobe研究院);Monash University(蒙纳士大学)

Xiangxi Shi, Xu Yang, Jiuxiang Gu, Shafiq Joty, Jianfei Cai

摘要:

改变字幕是一项旨在用自然语言描述图像之间差异的任务。大多数现有的方法把这个问题作为一个差异判断,而不存在干扰,如观点的变化。然而,在实践中,视点变化经常发生,并且可以压倒要描述的语义差异。在本文中,我们提出了一种新的视觉编码器,明确区分视点变化的语义变化中的变化字幕任务。此外,我们进一步模拟人类的注意力偏好,并提出了一种新的强化学习过程来微调的注意力直接与语言评价奖励。大量的实验结果表明,我们的方法优于国家的最先进的方法在两个斑点的差异和CLEVR-Change数据集的大幅度。

英文摘要:

Change Captioning is a task that aims to describe the difference between images with natural language. Most existing methods treat this problem as a difference judgment without the existence of distractors, such as viewpoint changes. However, in practice, viewpoint changes happen often and can overwhelm the semantic difference to be described. In this paper, we propose a novel visual encoder to explicitly distinguish viewpoint changes from semantic changes in the change captioning task. Moreover, we further simulate the attention preference of humans and propose a novel reinforcement learning process to fine-tune the attention directly with language evaluation rewards. Extensive experimental results show that our method outperforms the state-of-the-art approaches by a large margin in both Spot-the-Diff and CLEVR-Change datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑