This paper presents a new approach to form-filling by reformulating the task as multimodal natural language Question Answering (QA). The reformulation is achieved by first translating the elements on the GUI form (text fields, buttons, icons, etc.) to natural language questions, where these questions capture the element's multimodal semantics. After a match is determined between the form element (Question) and the user utterance (Answer), the form element is filled through a pre-trained extractive QA system. By leveraging pre-trained QA models and not requiring form-specific training, this approach to form-filling is zero-shot. The paper also presents an approach to further refine the form-filling by using multi-task training to incorporate a potentially large number of successive tasks. Finally, the paper introduces a multimodal natural language form-filling dataset Multimodal Forms (mForms), as well as a multimodal extension of the popular ATIS dataset to support future research and experimentation. Results show the new approach not only maintains robust accuracy for sparse training conditions but achieves state-of-the-art F1 of 0.97 on ATIS with approximately 1/10th of the training data.
The goal of Question Answering over Knowledge Graphs (KGQA) is to find answers for natural language questions over a knowledge graph. Recent KGQA approaches adopt a neural machine translation (NMT) approach, where the natural language question is translated into a structured query language. However, NMT suffers from the out-of-vocabulary problem, where terms in a question may not have been seen during training, impeding their translation. This issue is particularly problematic for the millions of entities that large knowledge graphs describe. We rather propose a KGQA approach that delegates th…
Arizona State University（亚利桑那州立大学）；University of Maryland, College Park（马里兰大学学院市分校）
In this work, we explore a genre of puzzles ("image riddles") which involves a set of images and a question. Answering these puzzles require both capabilities involving visual detection (including object, activity recognition) and, knowledge-based or commonsense reasoning. We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer. The annotations are validated using crowd-sourced evaluation. We also define an automatic evaluation metric to track future progress. Our task bears similarity with the commonly known IQ tasks such as analogy solving, seq…
In this paper, we present a system to showcase the capabilities of the latest state-of-the-art retrieval augmented generation models trained on knowledge-intensive language tasks, such as slot filling, open domain question answering, dialogue, and fact-checking. Moreover, given a user query, we show how the output from these different models can be combined to cross-examine the outputs of each other. Particularly, we show how accuracy in dialogue can be improved using the question answering model. We are also releasing all models used in the demo as a contribution of this paper. A short video …
Facebook AI Research（脸书人工智能研究院）；University College London（伦敦大学学院）；LORIA（洛林计算机科学及其应用实验室）；University of Cambridge（剑桥大学）；HuggingFace（HuggingFace公司）；University of Amsterdam（阿姆斯特丹大学）
Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot o…
Query graph construction aims to construct the correct executable SPARQL on the KG to answer natural language questions. Although recent methods have achieved good results using neural network-based query graph ranking, they suffer from three new challenges when handling more complex questions: 1) complicated SPARQL syntax, 2) huge search space, and 3) locally ambiguous query graphs. In this paper, we provide a new solution. As a preparation, we extend the query graph by treating each SPARQL clause as a subgraph consisting of vertices and edges and define a unified graph grammar called AQG to …
AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages
Mila Quebec AI Institute（魁北克AI研究所（Mila））；Research Center of Intelligent Machines, McGill University（麦吉尔大学智能机器研究中心）；Lelapa AI；GalsenAI；University of Zambia（赞比亚大学）；Ai4Innovr；RIKEN Center for AIP（理化学研究所AIP中心）；Lancaster University（兰卡斯特大学）；South African Centre for Digital Language Resources（南非数字语言资源中心）；Technical University of Munich（慕尼黑工业大学）；Jülich Supercomputing Centre（于利希超级计算中心）；Instituto Politécnico Nacional, Mexico（墨西哥国立理工学院）；Montclair State University（蒙特克莱尔州立大学）；University of Washington（华盛顿大学）；University of Porto,Insight Centre for Data Analytics（波尔图大学、数据洞察分析中心）；University of Amsterdam（阿姆斯特丹大学）；Accra Institute of Technology（阿克拉理工学院）；Kwame Nkrumah University of Science and Technology,University of Ibadan（夸梅·恩克鲁玛科技大学、伊巴丹大学）；Tom Mboya University（汤姆·博亚大学）
Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Pattern Recognition Center, WeChat AI（微信AI模式识别中心）
Sun Yat-sen University（中山大学）；East China Normal University（华东师范大学）；Columbia University（哥伦比亚大学）
Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspire…
Understanding inferences and answering questions from text requires more than merely recovering surface arguments, adjuncts, or strings associated with the query terms. As humans, we interpret sentences as contextualized components of a narrative or discourse, by both filling in missing information, and reasoning about event consequences. In this paper, we define the process of rewriting a textual expression (lexeme or phrase) such that it reduces ambiguity while also making explicit the underlying semantics that is not (necessarily) expressed in the economy of sentence structure as Dense Para…
Univ. of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究院）
We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot. This reduction has several advantages: we can (1) learn relation-extraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relation types that are only specified at test-time, for which we have no labeled training…
