arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.10328cs.CLcs.CV
收藏

R $^3$ Net:一种面向变化字幕的嵌入式表示重构网络

R$^3$Net:Relation-embedded Representation Reconstruction Network for Change Captioning

Kunming University of Science and Technology(昆明理工大学);Hangzhou Dianzi University(杭州电子科技大学)

Yunbin Tu, Liang Li, Chenggang Yan, Shengxiang Gao, Zhengtao Yu

摘要:

改变字幕是用一个自然语言句子来描述两个相似图像之间细粒度的不一致。视点变化是这一任务中最典型的干扰物,因为它改变了物体的尺度和位置,并使真实变化的表征变得模糊。在本文中,我们提出了一个嵌入式表示重建网络(R $^3$ Net),明确区分真正的变化,从大量的杂波和不相关的变化。具体来说,首先设计了一个关系嵌入模块,以探索大量的杂波中潜在的变化对象。然后,基于两幅图像中对应位置的语义相似性,设计了一个表征重构模块(RRM),用于学习重构表征,并进一步对差异表征进行建模。此外,我们引入了一个句法骨架预测器(SSP),以增强变化定位和字幕生成之间的语义交互。实验结果表明,该方法在两个公开数据集上取得了较好的效果。

英文摘要:

Change captioning is to use a natural language sentence to describe the fine-grained disagreement between two similar images. Viewpoint change is the most typical distractor in this task, because it changes the scale and location of the objects and overwhelms the representation of real change. In this paper, we propose a Relation-embedded Representation Reconstruction Network (R$^3$Net) to explicitly distinguish the real change from the large amount of clutter and irrelevant changes. Specifically, a relation-embedded module is first devised to explore potential changed objects in the large amount of clutter. Then, based on the semantic similarities of corresponding locations in the two images, a representation reconstruction module (RRM) is designed to learn the reconstruction representation and further model the difference representation. Besides, we introduce a syntactic skeleton predictor (SSP) to enhance the semantic interaction between change localization and caption generation. Extensive experiments show that the proposed method achieves the state-of-the-art results on two public datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑