Proceedings of the Workshop on Artificial Intelligence Safety (SafeAI 2020) co-located with 34th AAAI Conference on Artificial Intelligence (AAAI 2020), New York, USA, Feb 7, 2020
Explainability and interpretability of AI models is an essential factor affecting the safety of AI. While various explainable AI (XAI) approaches aim at mitigating the lack of transparency in deep networks, the evidence of the effectiveness of these approaches in improving usability, trust, and understanding of AI systems are still missing. We evaluate multimodal explanations in the setting of a Visual Question Answering (VQA) task, by asking users to predict the response accuracy of a VQA agent with and without explanations. We use between-subjects and within-subjects experiments to probe explanation effectiveness in terms of improving user prediction accuracy, confidence, and reliance, among other factors. The results indicate that the explanations help improve human prediction accuracy, especially in trials when the VQA system's answer is inaccurate. Furthermore, we introduce active attention, a novel method for evaluating causal attentional effects through intervention by editing attention maps. User explanation ratings are strongly correlated with human prediction accuracy and suggest the efficacy of these explanations in human-machine AI collaboration tasks.
EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents
The Australian National University（澳大利亚国立大学）；Royal Melbourne Institute of Technology（皇家墨尔本理工学院）；University of Sydney（悉尼大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）
PaniniQA: Enhancing Patient Education Through Interactive Question Answering
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Emory University（埃默里大学）；Northeastern University（东北大学）；UMass Chan Medical School（马萨诸塞大学陈医学院）；University of Minnesota（明尼苏达大学）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Textbook Question Answering (TQA) is a complex multimodal task to infer answers given large context descriptions and abundant diagrams. Compared with Visual Question Answering (VQA), TQA contains a large number of uncommon terminologies and various diagram inputs. It brings new challenges to the representation capability of language model for domain-specific spans. And it also pushes the multimodal fusion to a more complex level. To tackle the above issues, we propose a novel model named MoCA, which incorporates multi-stage domain pretraining and multimodal cross attention for the TQA task. Fi…
UC Berkeley EECS（加州大学伯克利分校电子工程与计算机科学系）；Sony Corp.（索尼公司）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality,…
Building AI agents can be costly. Consider a question answering agent such as Jill Watson that automatically answers students' questions on the discussion forums of online classes based on their syllabi and other course materials. Training a Jill on the syllabus of a new online class can take a hundred hours or more. Machine teaching - interactive teaching of an AI agent using synthetic data sets - can reduce the training time because it combines the advantages of knowledge-based AI, machine learning using large data sets, and interactive human-in-loop training. We describe Agent Smith, an int…
Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering
Faculty of Computing and Information Technology, King Abdulaziz University（计算机与信息科技学院，国王阿卜杜勒阿齐兹大学）；College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University (IMSIU)（计算机与信息科学学院，伊玛目穆罕默德·本·萨乌德伊斯兰大学）；School of Computing, Macquarie University（计算学院，麦考瑞大学）
Carnegie Mellon University（卡内基梅隆大学）；Waseda University（早稻田大学）；Yahoo Japan Corporation（雅虎日本公司）
Question categorization and expert retrieval methods have been crucial for information organization and accessibility in community question & answering (CQA) platforms. Research in this area, however, has dealt with only the text modality. With the increasing multimodal nature of web content, we focus on extending these methods for CQA questions accompanied by images. Specifically, we leverage the success of representation learning for text and images in the visual question answering (VQA) domain, and adapt the underlying concept and architecture for automated category classification and exper…
M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence（人工智能交叉学科项目）；Department of Intelligence and Information（智能与信息系）；Samsung Advanced Institute of Technology（三星先进技术研究院）
Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；NiuTrans Research（NiuTrans研究院）；School of Software, Northeastern University（东北大学软件学院）
MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering
Yale University, New Haven, CT, USA（耶鲁大学）；McGill University, Montreal, QC, Canada（麦吉尔大学）；University of Texas Rio Grande Valley, TX, USA（德克萨斯大学里奥格兰德谷分校）
Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
College of Computer Science, Chongqing University（重庆大学计算机学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Kuaishou Technology（快手科技）
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Shenzhen Institute of Advanced Technology Chinese Academy of Sciences（深圳先进技术研究院中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Dalian University of Technology（大连理工大学）；UNSW Sydney（悉尼大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
The Allen Institute for AI（艾伦人工智能研究所）；Tel-Aviv University（特拉维夫大学）；University of Washington（华盛顿大学）
When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources. While interest in models that reason over multiple pieces of evidence has surged in recent years, there has been relatively little work on question answering models that reason across multiple modalities. In this paper, we present MultiModalQA(MMQA): a challenging question answering dataset that requires joint reasoning over text, tables and images. We create MMQA using a new framework for generating complex multi-modal questions at scale, harvesting tables from Wikipedia, and …
Clova AI Research, NAVER Corp（NAVER公司Clova AI研究中心）；Seoul National University（首尔大学）；SK T-brain；Surromind Robotics
We propose a video story question-answering (QA) architecture, Multimodal Dual Attention Memory (MDAM). The key idea is to use a dual attention mechanism with late fusion. MDAM uses self-attention to learn the latent concepts in scene frames and captions. Given a question, MDAM uses the second attention over these latent concepts. Multimodal fusion is performed after the dual attention processes (late fusion). Using this processing pipeline, MDAM learns to infer a high-level vision-language joint representation from an abstraction of the full video content. We evaluate MDAM on PororoQA and Mov…
University of Bristol（布里斯托大学）；Royal Holloway, University of London（伦敦大学皇家霍洛威学院）
For many NLP applications, such as question answering and summarisation, the goal is to select the best solution from a large space of candidates to meet a particular user's needs. To address the lack of user-specific training data, we propose an interactive text ranking approach that actively selects pairs of candidates, from which the user selects the best. Unlike previous strategies, which attempt to learn a ranking across the whole candidate space, our method employs Bayesian optimisation to focus the user's labelling effort on high quality candidates and integrates prior knowledge in a Ba…
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
OPPO AI Center, OPPO Inc. China（OPPO AI中心，OPPO公司）；The Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Nanyang Technological University, Singapore（新加坡南洋理工大学）
