通过预测结构化概念改进图像描述生成
Improving Image Captioning via Predicting Structured Concepts
AI总结：
本文提出结构化概念预测器SCP和加权图卷积网络W-GCN，通过建模概念间关系并区分其贡献来改进图像描述生成。
摘要：
由于图像描述任务中难以解决图像与文本之间的语义鸿沟，该领域的传统研究将语义概念视为两种模态之间的桥梁，并据此提升了描述生成性能。尽管在概念预测上取得了有前景的结果，上述研究通常忽略了概念之间的关系，而这种关系不仅依赖于图像中的物体，还依赖于文本中的词语依赖，因此为改进高质量描述生成过程提供了相当大的潜力。本文提出了一种结构化概念预测器（SCP）来预测概念及其结构，然后将其集成到描述生成中，从而通过概念增强视觉信号在该任务中的贡献，并进一步利用概念之间的关系来区分跨模态语义，以生成更好的描述。特别地，我们设计了加权图卷积网络（W-GCN）来刻画由词语依赖驱动的概念关系，并学习这些概念对后续解码过程的差异化贡献。因此，我们的方法捕捉了概念之间的潜在关系，并有区分地学习不同概念，从而通过跨模态继承的信息有效促进图像描述生成。大量实验及其结果证明了本文方法以及所提出的每个模块的有效性。
英文摘要：
Having the difficulty of solving the semantic gap between images and texts for the image captioning task, conventional studies in this area paid some attention to treating semantic concepts as a bridge between the two modalities and improved captioning performance accordingly. Although promising results on concept prediction were obtained, the aforementioned studies normally ignore the relationship among concepts, which relies on not only objects in the image, but also word dependencies in the text, so that offers a considerable potential for improving the process of generating good descriptions. In this paper, we propose a structured concept predictor (SCP) to predict concepts and their structures, then we integrate them into captioning, so as to enhance the contribution of visual signals in this task via concepts and further use their relations to distinguish cross-modal semantics for better description generation. Particularly, we design weighted graph convolutional networks (W-GCN) to depict concept relations driven by word dependencies, and then learns differentiated contributions from these concepts for following decoding process. Therefore, our approach captures potential relations among concepts and discriminatively learns different concepts, so that effectively facilitates image captioning with inherited information across modalities. Extensive experiments and their results demonstrate the effectiveness of our approach as well as each proposed module in this work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
