arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.08322cs.CVcs.MM
收藏

ORD:面向可视对话生成的对象关系发现

ORD: Object Relationship Discovery for Visual Dialogue Generation

The University of Queensland(昆士兰大学);Kyushu Institute of Technology(九州工业大学)

Ziwei Wang, Zi Huang, Yadan Luo, Huimin Lu

摘要:

随着图像描述和视觉问答在单轮水平上的快速推进,如何生成关于视觉内容的多轮对话的问题尚未得到很好的探索,现有的视觉对话方法直接将图像编码为固定的特征向量,与问题和历史嵌入连接起来,以预测响应。最近的一些方法使用共同参考解决问题,注意力机制可以交叉引用图像、历史和目标问题中的相关元素。然而,由于在共同注意推理之前忽略了细粒度的对象级信息,因此推理视觉关系仍然具有挑战性。在本文中,我们提出了一个对象关系发现(ORD)框架,以保持视觉对话生成的对象交互。具体地说,层次图卷积网络(HierGCN)被提出来保留对象节点和邻居关系的本地,然后细化对象对象之间的连接,以获得最终的图嵌入。一个图形的注意力进一步纳入动态出席这个图形结构的表示在响应推理阶段。大量的实验证明,该方法可以显着提高质量的对话,利用视觉关系的上下文信息。该模型在Visual Dialog数据集上实现了优于最先进方法的性能,将MRR从0.6222提高到0.6447,recall@1从48.48%提高到51.22%。

英文摘要:

With the rapid advancement of image captioning and visual question answering at single-round level, the question of how to generate multi-round dialogue about visual content has not yet been well explored.Existing visual dialogue methods encode the image into a fixed feature vector directly, concatenated with the question and history embeddings to predict the response.Some recent methods tackle the co-reference resolution problem using co-attention mechanism to cross-refer relevant elements from the image, history, and the target question.However, it remains challenging to reason visual relationships, since the fine-grained object-level information is omitted before co-attentive reasoning. In this paper, we propose an object relationship discovery (ORD) framework to preserve the object interactions for visual dialogue generation. Specifically, a hierarchical graph convolutional network (HierGCN) is proposed to retain the object nodes and neighbour relationships locally, and then refines the object-object connections globally to obtain the final graph embeddings. A graph attention is further incorporated to dynamically attend to this graph-structured representation at the response reasoning stage. Extensive experiments have proved that the proposed method can significantly improve the quality of dialogue by utilising the contextual information of visual relationships. The model achieves superior performance over the state-of-the-art methods on the Visual Dialog dataset, increasing MRR from 0.6222 to 0.6447, and recall@1 from 48.48% to 51.22%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑