Visual dialog is a task of answering a sequence of questions grounded in an image using the previous dialog history as context. In this paper, we study how to address two fundamental challenges for this task: (1) reasoning over underlying semantic structures among dialog rounds and (2) identifying several appropriate answers to the given question. To address these challenges, we propose a Sparse Graph Learning (SGL) method to formulate visual dialog as a graph structure learning task. SGL infers inherently sparse dialog structures by incorporating binary and score edges and leveraging a new structural loss function. Next, we introduce a Knowledge Transfer (KT) method that extracts the answer predictions from the teacher model and uses them as pseudo labels. We propose KT to remedy the shortcomings of single ground-truth labels, which severely limit the ability of a model to obtain multiple reasonable answers. As a result, our proposed model significantly improves reasoning capability compared to baseline methods and outperforms the state-of-the-art approaches on the VisDial v1.0 dataset. The source code is available at https://github.com/gicheonkang/SGLKT-VisDial.
Singapore Management University（新加坡管理大学）；A*STAR, Institute for Infocomm Research（新加坡科技研究局资讯通信研究院）；Salesforce Research Asia（赛富时亚洲研究院）
Compared to traditional visual question answering, video-grounded dialogues require additional reasoning over dialogue context to answer questions in a multi-turn setting. Previous approaches to video-grounded dialogues mostly use dialogue context as a simple text input without modelling the inherent information flows at the turn level. In this paper, we propose a novel framework of Reasoning Paths in Dialogue Context (PDC). PDC model discovers information flows among dialogue turns through a semantic graph constructed based on lexical components in each question and answer. PDC model then lea…
Reasoning Visual Dialogs with Structural and Partial Observations
University of California, Los Angeles（加州大学洛杉矶分校）；Inception Institute of Artificial Intelligence（人工智能初创研究院）；International Center for AI and Robot Autonomy (CARA)（国际人工智能与机器人自主中心（CARA））
Applied Artificial Intelligence Institute, Deakin University（迪肯大学应用人工智能研究所）；Posts and Telecommunications Institute of Technology（邮电技术学院）
It would be a technological feat to be able to create a system that can hold a meaningful conversation with humans about what they watch. A setup toward that goal is presented as a video dialog task, where the system is asked to generate natural utterances in response to a question in an ongoing dialog. The task poses great visual, linguistic, and reasoning challenges that cannot be easily overcome without an appropriate representation scheme over video and dialog that supports high-level reasoning. To tackle these challenges we present a new object-centric framework for video dialog that supp…
Visual Dialog requires an agent to engage in a conversation with humans grounded in an image. Many studies on Visual Dialog focus on the understanding of the dialog history or the content of an image, while a considerable amount of commonsense-required questions are ignored. Handling these scenarios depends on logical reasoning that requires commonsense priors. How to capture relevant commonsense knowledge complementary to the history and the image remains a key challenge. In this paper, we propose a novel model by Reasoning with Multi-structure Commonsense Knowledge (RMK). In our model, the e…
Beihang University（北京航空航天大学）；Codemao Inc.（编程猫公司）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Visual dialogue is a challenging task that needs to extract implicit information from both visual (image) and textual (dialogue history) contexts. Classical approaches pay more attention to the integration of the current question, vision knowledge and text knowledge, despising the heterogeneous semantic gaps between the cross-modal information. In the meantime, the concatenation operation has become de-facto standard to the cross-modal information fusion, which has a limited ability in information retrieval. In this paper, we propose a novel Knowledge-Bridge Graph Network (KBGN) model by using…
Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；Pengcheng Laboratory（鹏城实验室）；School of Computer Science and Technology, Shandong Jianzhu University（山东建筑大学计算机科学与技术学院）
Rutgers University（罗格斯大学）；The Chinese University of Hong Kong（香港中文大学）；University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
Given an input video, its associated audio, and a brief caption, the audio-visual scene aware dialog (AVSD) task requires an agent to indulge in a question-answer dialog with a human about the audio-visual content. This task thus poses a challenging multi-modal representation learning and reasoning scenario, advancements into which could influence several human-machine interaction applications. To solve this task, we introduce a semantics-controlled multi-modal shuffled Transformer reasoning framework, consisting of a sequence of Transformer modules, each taking a modality as input and produci…
DialogueVPR: Towards Conversational Visual Place Recognition
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Shandong Computer Science Center, Qilu University of Technology（山东省计算中心（国家超级计算济南中心），齐鲁工业大学）；Macquarie University（麦考瑞大学）；Spatialtemporal AI（时空人工智能公司）；University of Macau（澳门大学）；Aerospace Information Research Institute（航天信息研究所）
Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memory
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Jianghuai Advance Technology Center（江淮先进技术中心）；Anhui Provincial Key Laboratory of Humanoid Robots（安徽省人形机器人重点实验室）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；University of Macau（澳门大学）
In this work, we formulate a visual dialog as an information flow in which each piece of information is encoded with the joint visual-linguistic representation of a single dialog round. Based on this formulation, we consider the visual dialog task as a sequence problem consisting of ordered visual-linguistic vectors. For featurization, we use a Dense Symmetric Co-Attention network as a lightweight vison-language joint representation generator to fuse multimodal features (i.e., image and text), yielding better computation and data efficiencies. For inference, we propose two Sequential Dialog Ne…
The University of Queensland（昆士兰大学）；Kyushu Institute of Technology（九州工业大学）
With the rapid advancement of image captioning and visual question answering at single-round level, the question of how to generate multi-round dialogue about visual content has not yet been well explored.Existing visual dialogue methods encode the image into a fixed feature vector directly, concatenated with the question and history embeddings to predict the response.Some recent methods tackle the co-reference resolution problem using co-attention mechanism to cross-refer relevant elements from the image, history, and the target question.However, it remains challenging to reason visual relati…
