arXiv 2007.033102020-07-07cs.CVcs.CL

DAM：视觉对话中产生详细和非重复反应的思考、放弃和记忆网络

DAM: Deliberation, Abandon and Memory Networks for Generating Detailed and Non-repetitive Responses in Visual Dialogue

Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Beihang University（北京航空航天大学）；University of Chinese Academy of Sciences（中国科学院大学）；Codemao Inc.（编程猫公司）；University of Adelaide（阿德莱德大学）

Xiaoze Jiang, Jing Yu, Yajing Sun, Zengchang Qin, Zihao Zhu, Yue Hu, Qi Wu

备注：: Accepted by IJCAI 2020. SOLE copyright holder is IJCAI (International Joint Conferences on Artificial Intelligence)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

摘要：

视觉对话任务要求智能体参与与人类关于图像的对话。生成详细和非重复响应的能力对于Agent实现类似人类的对话至关重要。在本文中，我们提出了一种新的生成解码架构，以生成高质量的响应，远离解码整个编码语义的设计，提倡透明性和灵活性。在这个架构中，单词生成被分解成一系列的注意力为基础的信息选择步骤，由新的经常性审议，放弃和记忆（DAM）模块执行。每个DAM模块执行从编码器捕获的响应级语义和专门为生成每个单词而选择的单词级语义的自适应组合。因此，响应包含更详细和非重复的描述，同时保持语义准确性。此外，DAM是灵活的，以配合现有的视觉对话编码器和自适应的编码器结构，通过限制在DAM的信息选择模式。我们将DAM应用于三个典型的编码器，并在VisDial v1.0数据集上验证了性能。实验结果表明，所提出的模型实现了新的国家的最先进的性能与高质量的响应。该代码可在www.example.com上获得。

英文摘要：

Visual Dialogue task requires an agent to be engaged in a conversation with human about an image. The ability of generating detailed and non-repetitive responses is crucial for the agent to achieve human-like conversation. In this paper, we propose a novel generative decoding architecture to generate high-quality responses, which moves away from decoding the whole encoded semantics towards the design that advocates both transparency and flexibility. In this architecture, word generation is decomposed into a series of attention-based information selection steps, performed by the novel recurrent Deliberation, Abandon and Memory (DAM) module. Each DAM module performs an adaptive combination of the response-level semantics captured from the encoder and the word-level semantics specifically selected for generating each word. Therefore, the responses contain more detailed and non-repetitive descriptions while maintaining the semantic accuracy. Furthermore, DAM is flexible to cooperate with existing visual dialogue encoders and adaptive to the encoder structures by constraining the information selection mode in DAM. We apply DAM to three typical encoders and verify the performance on the VisDial v1.0 dataset. Experimental results show that the proposed models achieve new state-of-the-art performance with high-quality responses. The code is available at https://github.com/JXZe/DAM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。