The ability to ask questions is important in both human and machine intelligence. Learning to ask questions helps knowledge acquisition, improves question-answering and machine reading comprehension tasks, and helps a chatbot to keep the conversation flowing with a human. Existing question generation models are ineffective at generating a large amount of high-quality question-answer pairs from unstructured text, since given an answer and an input passage, question generation is inherently a one-to-many mapping. In this paper, we propose Answer-Clue-Style-aware Question Generation (ACS-QG), which aims at automatically generating high-quality and diverse question-answer pairs from unlabeled text corpus at scale by imitating the way a human asks questions. Our system consists of: i) an information extractor, which samples from the text multiple types of assistive information to guide question generation; ii) neural question generators, which generate diverse and controllable questions, leveraging the extracted assistive information; and iii) a neural quality controller, which removes low-quality generated data based on text entailment. We compare our question generation models with existing approaches and resort to voluntary human evaluation to assess the quality of the generated question-answer pairs. The evaluation results suggest that our system dramatically outperforms state-of-the-art neural question generation models in terms of the generation quality, while being scalable in the meantime. With models trained on a relatively smaller amount of data, we can generate 2.8 million quality-assured question-answer pairs from a million sentences found in Wikipedia.
This paper describes a method for asking statistical questions about a large text corpus. We exemplify the method by addressing the question, "What percentage of Federal Register documents are real documents, of possible interest to a text researcher or analyst?" We estimate an answer to this question by evaluating 200 documents selected from a corpus of 45,820 Federal Register documents. Stratified sampling is used to reduce the sampling uncertainty of the estimate from over 3100 documents to fewer than 1000. The stratification is based on observed characteristics of real documents, while the…
University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Microsoft Research, Montréal（微软蒙特利尔研究院）；Google AI, Mountain View（谷歌山景城人工智能部门）
This paper introduces a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other. The framework is agnostic to the architecture of the machine reading model, only requiring access to the token-level hidden representations of the reader. The retriever uses fast nearest neighbor search to scale to corpora containing millions of paragraphs. A gated recurrent unit updates the query at each step conditioned on the state of the reader and the reformulated query is used to re-rank the paragraphs by the retriever. We conduct analysis a…
SkillFlow: Scalable and Efficient Agent Skill Retrieval System
University of California, Davis（加州大学戴维斯分校）；USDA/NSF AI Institute for Next Generation Food Systems（美国农业部/国家科学基金会下一代食品系统人工智能研究所）；Process Integration and Predictive Analytics（过程集成与预测分析）
BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms
University of Science and Technology of China（中国科学技术大学）；Metastone Technology（Metastone科技）；Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences（北京农林科学院信息技术研究中心）
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Information seeking users often pose questions with false presuppositions, especially when asking about unfamiliar topics. Most existing question answering (QA) datasets, in contrast, assume all questions have well defined answers. We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums. We find that 25% of questions contain false presuppositions, and provide annotations for these presuppositions and their corrections. Through extensive baseline experiments, we show that adaptations of existing open-domain QA models c…
U.S. Army Research Laboratory（美国陆军研究实验室）；Tufts University（塔夫茨大学）；U.S. Naval Research Laboratory（美国海军研究实验室）
Intelligent agents that are confronted with novel concepts in situated environments will need to ask their human teammates questions to learn about the physical world. To better understand this problem, we need data about asking questions in situated task-based interactions. To this end, we present the Human-Robot Dialogue Learning (HuRDL) Corpus - a novel dialogue corpus collected in an online interactive virtual environment in which human participants play the role of a robot performing a collaborative tool-organization task. We describe the corpus data and a corresponding annotation scheme …
University of Massachusetts（马萨诸塞大学）；Megagon Labs（Megagon实验室）；Google Japan（谷歌日本）
We present the KOKO system that takes declarative information extraction to a new level by incorporating advances in natural language processing techniques in its extraction language. KOKO is novel in that its extraction language simultaneously supports conditions on the surface of the text and on the structure of the dependency parse tree of sentences, thereby allowing for more refined extractions. KOKO also supports conditions that are forgiving to linguistic variation of expressing concepts and allows to aggregate evidence from the entire document in order to filter extractions. To scale up…
University of Manchester（曼彻斯特大学）；Idiap Research Institute（Idiap研究所）
This paper presents a novel framework for reconstructing multi-hop explanations in science Question Answering (QA). While existing approaches for multi-hop reasoning build explanations considering each question in isolation, we propose a method to leverage explanatory patterns emerging in a corpus of scientific explanations. Specifically, the framework ranks a set of atomic facts by integrating lexical relevance with the notion of unification power, estimated analysing explanations for similar questions in the corpus. An extensive evaluation is performed on the Worldtree corpus, integrating k-…
MAQuA: Adaptive Question-Asking for Multidimensional Mental Health Screening using Item Response Theory
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Stony Brook University（石溪大学）；Aarhus University（奥胡斯大学）；Department of Psychology, Lund University（吕勒欧大学心理学系）；College of Connected Computing, Vanderbilt University（范德比尔特大学连接计算学院）
Popular QA benchmarks like SQuAD have driven progress on the task of identifying answer spans within a specific passage, with models now surpassing human performance. However, retrieving relevant answers from a huge corpus of documents is still a challenging problem, and places different requirements on the model architecture. There is growing interest in developing scalable answer retrieval models trained end-to-end, bypassing the typical document retrieval step. In this paper, we introduce Retrieval Question-Answering (ReQA), a benchmark for evaluating large-scale sentence-level answer retri…
Corpus Prevalence of Multiple-Choice Question Options
Center for Language and Cognition, University of Groningen（语言与认知中心，格罗宁根大学）；Department of Experimental Psychology, University of Groningen（实验心理学系，格罗宁根大学）
The paper presents an approach to build a question and answer system that is capable of processing the information in a large dataset and allows the user to gain knowledge from this dataset by asking questions in natural language form. Key content of this research covers four dimensions which are; Corpus Preprocessing, Question Preprocessing, Deep Neural Network for Answer Extraction and Answer Generation. The system is capable of understanding the question, responds to the user's query in natural language form as well. The goal is to make the user feel as if they were interacting with a perso…
The most prominent tasks in emotion analysis are to assign emotions to texts and to understand how emotions manifest in language. An observation for NLP is that emotions can be communicated implicitly by referring to events, appealing to an empathetic, intersubjective understanding of events, even without explicitly mentioning an emotion name. In psychology, the class of emotion theories known as appraisal theories aims at explaining the link between events and emotions. Appraisals can be formalized as variables that measure a cognitive evaluation by people living through an event that they co…
Beijing University of Posts and Telecommunications（北京邮电大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft STCA（微软STCA）；University of Science and Technology of China（中国科学技术大学）
Ad-hoc search calls for the selection of appropriate answers from a massive-scale corpus. Nowadays, the embedding-based retrieval (EBR) becomes a promising solution, where deep learning based document representation and ANN search techniques are allied to handle this task. However, a major challenge is that the ANN index can be too large to fit into memory, given the considerable size of answer corpus. In this work, we tackle this problem with Bi-Granular Document Representation, where the lightweight sparse embeddings are indexed and standby in memory for coarse-grained candidate search, and …
Given a small corpus $\mathcal D_T$ pertaining to a limited set of focused topics, our goal is to train embeddings that accurately capture the sense of words in the topic in spite of the limited size of $\mathcal D_T$. These embeddings may be used in various tasks involving $\mathcal D_T$. A popular strategy in limited data settings is to adapt pre-trained embeddings $\mathcal E$ trained on a large corpus. To correct for sense drift, fine-tuning, regularization, projection, and pivoting have been proposed recently. Among these, regularization informed by a word's corpus frequency performed wel…
University of Illinois College of Medicine（伊利诺伊大学医学院）
Neural embeddings are a popular set of methods for representing words, phrases or text as a low dimensional vector (typically 50-500 dimensions). However, it is difficult to interpret these dimensions in a meaningful manner, and creating neural embeddings requires extensive training and tuning of multiple parameters and hyperparameters. We present here a simple unsupervised method for representing words, phrases or text as a low dimensional vector, in which the meaning and relative importance of dimensions is transparent to inspection. We have created a near-comprehensive vector representation…
Technical University of Munich（慕尼黑工业大学）；Loyola University Chicago（芝加哥洛约拉大学）
We describe a method and new no-code software tools enabling domain experts to build custom structured, labeled datasets from the unstructured text of documents and build niche machine learning text classification models traceable to expert-written rules. The Context Rule Assisted Machine Learning (CRAML) method allows accurate and reproducible labeling of massive volumes of unstructured text. CRAML enables domain experts to access uncommon constructs buried within a document corpus, and avoids limitations of current computational approaches that often lack context, transparency, and interpeta…
ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluation
Johns Hopkins University（约翰霍普金斯大学）；Liaoning Technical University（辽宁技术大学）；Microsoft Research Montréal（微软研究院蒙特利尔分校）；Central University of Finance and Economics（中央财经大学）
This paper proposes a first attempt to build an end-to-end speech-to-text translation system, which does not use source language transcription during learning or decoding. We propose a model for direct speech-to-text translation, which gives promising results on a small French-English synthetic corpus. Relaxing the need for source language transcription would drastically change the data collection methodology in speech translation, especially in under-resourced scenarios. For instance, in the former project DARPA TRANSTAC (speech translation from spoken Arabic dialects), a large effort was dev…
We propose Quootstrap, a method for extracting quotations, as well as the names of the speakers who uttered them, from large news corpora. Whereas prior work has addressed this problem primarily with supervised machine learning, our approach follows a fully unsupervised bootstrapping paradigm. It leverages the redundancy present in large news corpora, more precisely, the fact that the same quotation often appears across multiple news articles in slightly different contexts. Starting from a few seed patterns, such as ["Q", said S.], our method extracts a set of quotation-speaker pairs (Q, S), w…
C-VARC: A Large-Scale Chinese Value Rule Corpus for Value Alignment of Large Language Models
BrainCog Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所脑认知实验室，北京，中国）；Beijing Key Laboratory of Safe AI and Superalignment, Beijing, China（北京安全人工智能与超对齐关键实验室，北京，中国）；Beijing Institute of AI Safety and Governance, Beijing, China（北京人工智能安全与治理研究所，北京，中国）；The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院，北京，中国）；Long-term AI, Beijing, China（长期人工智能，北京，中国）
HACHIMI: Scalable and Controllable Student Persona Generation via Orchestrated Agents
East China Normal University（华东师范大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Shanghai Innovation Institute（上海创新研究院）
Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web
NOVA School of Science and Technology（NOVA科学与技术学院）；NOVA LINCS；Fundação para a Ciência e Tecnologia（科学技术基金会）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Instituto de Telecomunicações（电信研究院）
