The Visual Dialog task requires a model to exploit both image and conversational context information to generate the next response to the dialogue. However, via manual analysis, we find that a large number of conversational questions can be answered by only looking at the image without any access to the context history, while others still need the conversation context to predict the correct answers. We demonstrate that due to this reason, previous joint-modality (history and image) models over-rely on and are more prone to memorizing the dialogue history (e.g., by extracting certain keywords or patterns in the context information), whereas image-only models are more generalizable (because they cannot memorize or extract keywords from history) and perform substantially better at the primary normalized discounted cumulative gain (NDCG) task metric which allows multiple correct answers. Hence, this observation encourages us to explicitly maintain two models, i.e., an image-only model and an image-history joint model, and combine their complementary abilities for a more balanced multimodal model. We present multiple methods for this integration of the two models, via ensemble and consensus dropout fusion with shared parameters. Empirically, our models achieve strong results on the Visual Dialog challenge 2019 (rank 3 on NDCG and high balance across metrics), and substantially outperform the winner of the Visual Dialog challenge 2018 on most metrics.
Academia Sinica（中央研究院）；Cornell Tech（康奈尔理工学院）；University of Colorado Boulder（科罗拉多大学博尔德分校）
Understanding dynamic scenes and dialogue contexts in order to converse with users has been challenging for multimodal dialogue systems. The 8-th Dialog System Technology Challenge (DSTC8) proposed an Audio Visual Scene-Aware Dialog (AVSD) task, which contains multiple modalities including audio, vision, and language, to evaluate how dialogue systems understand different modalities and response to users. In this paper, we proposed a multi-step joint-modality attention network (JMAN) based on recurrent neural network (RNN) to reason on videos. Our model performs a multi-step attention mechanism…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent Cloud Xiaowei（腾讯云小微）
Existing text- and image-based multimodal dialogue systems use the traditional Hierarchical Recurrent Encoder-Decoder (HRED) framework, which has an utterance-level encoder to model utterance representation and a context-level encoder to model context representation. Although pioneer efforts have shown promising performances, they still suffer from the following challenges: (1) the interaction between textual features and visual features is not fine-grained enough. (2) the context representation can not provide a complete representation for the context. To address the issues mentioned above, w…
Dialogue state tracking (DST) is a pivotal component in task-oriented dialogue systems. While it is relatively easy for a DST model to capture belief states in short conversations, the task of DST becomes more challenging as the length of a dialogue increases due to the injection of more distracting contexts. In this paper, we aim to improve the overall performance of DST with a special focus on handling longer dialogues. We tackle this problem from three perspectives: 1) A model designed to enable hierarchical slot status prediction; 2) Balanced training procedure for generic and task-specifi…
The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods
Mines Paris-PSL University（巴黎高科矿业学院）；University of Bern（伯尔尼大学）；National Technical University of Athens（雅典国立技术大学）；Archimedes AI（阿基米德人工智能公司）；Synaptic Bloom PBC（Synaptic Bloom公益公司）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Beihang University（北京航空航天大学）；Microsoft Research Asia（微软亚洲研究院）；University of Adelaide（阿德莱德大学）
Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue involves multiple questions which cover a broad range of visual content that could be related to any objects, relationships or semantics. The key challenge in Visual Dialogue task is thus to learn a more comprehensive and semantic-rich image representation which may have adaptive attentions on the image for variant questions. In this research, we propose a novel model to depict an image from both visual and semantic perspectives. Specifically, the visual view helps capture th…
CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology (HUST)（华中科技大学人工智能与自动化学院）；Institute of Artificial Intelligence, HUST（华中科技大学人工智能研究院）；Hubei Key Laboratory of Brain-inspired Intelligent Systems, HUST（华中科技大学湖北省脑启发智能系统重点实验室）；Key Laboratory of Image Processing and Intelligent Control (HUST), Ministry of Education（华中科技大学图像信息处理与智能控制教育部重点实验室）
Academia Sinica（中央研究院）；National Taiwan University（国立台湾大学）
With increasing information from social media, there are more and more videos available. Therefore, the ability to reason on a video is important and deserves to be discussed. TheDialog System Technology Challenge (DSTC7) (Yoshino et al. 2018) proposed an Audio Visual Scene-aware Dialog (AVSD) task, which contains five modalities including video, dialogue history, summary, and caption, as a scene-aware environment. In this paper, we propose the entropy-enhanced dynamic memory network (DMN) to effectively model video modality. The attention-based GRU in the proposed model can improve the model'…
Multi-Turn Multi-Agent Dialogue for Collaborative Reconstruction Improves VLM Performance on Spatial Reasoning, But Only Barely
Computational Linguistics, Department of Linguistics University of Potsdam（语言学计算系，语言学系 柏林洪堡大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue
University of Chinese Academy of Sciences（中国科学院大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Huawei Technologies Ltd.（华为技术有限公司）；Beijing University of Post and Telecommunications（北京邮电大学）
Dual-Space Smoothness for Robust and Balanced LLM Unlearning
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Department of Computer Science and Engineering, University of Notre Dame（圣母大学计算机科学与工程系）
University of Amsterdam（阿姆斯特丹大学）；Microsoft Research（微软研究院）
Understanding audio-visual content and the ability to have an informative conversation about it have both been challenging areas for intelligent systems. The Audio Visual Scene-aware Dialog (AVSD) challenge, organized as a track of the Dialog System Technology Challenge 7 (DSTC7), proposes a combined task, where a system has to answer questions pertaining to a video given a dialogue with previous question-answer pairs and the video itself. We propose for this task a hierarchical encoder-decoder model which computes a multi-modal embedding of the dialogue context. It first embeds the dialogue h…
