Here we experiment with the use of information retrieval as an augmentation for pre-trained language models. The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time. By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training. We also validate our IR augmentation on an event co-reference task.
Neural retrievers based on pre-trained language models (PLMs), such as dual-encoders, have achieved promising performance on the task of open-domain question answering (QA). Their effectiveness can further reach new state-of-the-arts by incorporating cross-architecture knowledge distillation. However, most of the existing studies just directly apply conventional distillation methods. They fail to consider the particular situation where the teacher and student have different structures. In this paper, we propose a novel distillation method that significantly advances cross-architecture distilla…
On-the-fly Denoising for Data Augmentation in Natural Language Understanding
Hong Kong University of Science and Technology（香港科技大学）；University of Southern California（南加州大学）；University of Cambridge（剑桥大学）；Tencent AI Lab, Seattle（腾讯AI实验室（西雅图））；University of California, Davis（加州大学戴维斯分校）
Neural models that independently project questions and answers into a shared embedding space allow for efficient continuous space retrieval from large corpora. Independently computing embeddings for questions and answers results in late fusion of information related to matching questions to their answers. While critical for efficient retrieval, late fusion underperforms models that make use of early fusion (e.g., a BERT based classifier with cross-attention between question-answer pairs). We present a supervised data mining method using an accurate early fusion model to improve the training of…
University of Mannheim（曼海姆大学）；University of Verona（维罗纳大学）；University of Würzburg（维尔茨堡大学）
This paper introduces our proposed system for the MIA Shared Task on Cross-lingual Open-retrieval Question Answering (COQA). In this challenging scenario, given an input question the system has to gather evidence documents from a multilingual pool and generate from them an answer in the language of the question. We devised several approaches combining different model variants for three main components: Data Augmentation, Passage Retrieval, and Answer Generation. For passage retrieval, we evaluated the monolingual BM25 ranker against the ensemble of re-rankers based on multilingual pretrained l…
Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing
University of Michigan（密歇根大学）；University of British Columbia（不列颠哥伦比亚大学）；Rutgers University（罗格斯大学）；University of Pennsylvania（宾夕法尼亚大学）；New Jersey Institute of Technology（新泽西理工学院）；TU Darmstadt（图腾斯大学）；Wake Forest University（威克森林大学）
Cost-Aware Retrieval-Augmentation Reasoning Models with Adaptive Retrieval Depth
Microsoft（微软公司）；Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（巴黎-罗克奎恩特研究所）；Rajiv Gandhi University（拉贾夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒尔研究实验室）
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；University of Science and Technology of China（中国科学技术大学）
Deep neural retrieval models have amply demonstrated their power but estimating the reliability of their predictions remains challenging. Most dialog response retrieval models output a single score for a response on how relevant it is to a given question. However, the bad calibration of deep neural network results in various uncertainty for the single score such that the unreliable predictions always misinform user decisions. To investigate these issues, we present an efficient calibration and uncertainty estimation framework PG-DRR for dialog response retrieval models which adds a Gaussian Pr…
DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generation
School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥综合性国家科学中心）
Exploration of Augmentation Strategies in Multi-modal Retrieval-Augmented Generation for the Biomedical Domain: A Case Study Evaluating Question Answering in Glycobiology
University of Maribor, Faculty of Health Sciences（莫拉维亚大学健康科学学院）；University of Ljubljana, Medical Factory（卢布尔雅那大学医疗工厂）；Genos Ltd（基因公司）；Center for Smart Health, School of Nursing The Hong Kong Polytechnic University（智能健康中心护理学院香港理工大学）；University of Zagreb, Faculty of Pharmacy（扎格雷布大学药学院）；Usher Institute University of Edinburgh（埃德蒙顿大学usher研究所）
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Huawei Poisson Lab（华为泊松实验室）；Waseda University（早稻田大学）
Despite showing increasingly human-like conversational abilities, state-of-the-art dialogue models often suffer from factual incorrectness and hallucination of knowledge (Roller et al., 2020). In this work we explore the use of neural-retrieval-in-the-loop architectures - recently shown to be effective in open-domain QA (Lewis et al., 2020b; Izacard and Grave, 2020) - for knowledge-grounded dialogue, a task that is arguably more challenging as it requires querying based on complex multi-turn dialogue context and generating conversationally coherent responses. We study various types of architec…
Lab-AI: Using Retrieval Augmentation to Enhance Language Models for Personalized Lab Test Interpretation in Clinical Medicine
Department of Statistics（统计学系）；Florida State University（佛罗里达州立大学）；School of Information（信息学院）；Oklahoma State University（俄克拉荷马州立大学）；Morsani College of Medicine（医学学院）；University of South Florida（佛罗里达大学）；College of Social Work（社会工作学院）；Department of Computer Science（计算机科学系）；Emory University（埃默里大学）
CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation
Department of Computer Science, University of Copenhagen（哥本哈根大学计算机科学系）；Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
HiQA: A Hierarchical Contextual Augmentation RAG for Multi-Documents QA
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Southeast University（东南大学）；Hello World(Shanghai) Technology Co., Ltd.（Hello World（上海）科技有限公司）
Retrieval augmentation has shown promising improvements in different tasks. However, whether such augmentation can assist a large language model based re-ranker remains unclear. We investigate how to augment T5-based re-rankers using high-quality information retrieved from two external corpora -- a commercial web search engine and Wikipedia. We empirically demonstrate how retrieval augmentation can substantially improve the effectiveness of T5-based re-rankers for both in-domain and zero-shot out-of-domain re-ranking tasks.
In practical applications of semantic parsing, we often want to rapidly change the behavior of the parser, such as enabling it to handle queries in a new domain, or changing its predictions on certain targeted queries. While we can introduce new training examples exhibiting the target behavior, a mechanism for enacting such behavior changes without expensive model re-training would be preferable. To this end, we propose ControllAble Semantic Parser via Exemplar Retrieval (CASPER). Given an input query, the parser retrieves related exemplars from a retrieval index, augments them to the query, a…
Zeta Alpha（泽塔阿尔法）；NeuralMind（神经思维）；University of Campinas（坎皮纳斯大学）；University of Waterloo（滑铁卢大学）
The information retrieval community has recently witnessed a revolution due to large pretrained transformer models. Another key ingredient for this revolution was the MS MARCO dataset, whose scale and diversity has enabled zero-shot transfer learning to various tasks. However, not all IR tasks and domains can benefit from one single dataset equally. Extensive research in various NLP tasks has shown that using domain-specific training data, as opposed to a general-purpose one, improves the performance of neural models. In this work, we harness the few-shot capabilities of large pretrained langu…
Large Language Models (LLMs) have shown impressive results on a variety of text understanding tasks. Search queries though pose a unique challenge, given their short-length and lack of nuance or context. Complicated feature engineering efforts do not always lead to downstream improvements as their performance benefits may be offset by increased complexity of knowledge distillation. Thus, in this paper we make the following contributions: (1) We demonstrate that Retrieval Augmentation of queries provides LLMs with valuable additional context enabling improved understanding. While Retrieval Augm…
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Institute for AI, Tsinghua University（清华大学人工智能研究院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）
University of Waterloo（滑铁卢大学）；University of Washington（华盛顿大学）；Meta AI
Various techniques have been developed in recent years to improve dense retrieval (DR), such as unsupervised contrastive learning and pseudo-query generation. Existing DRs, however, often suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, which some argue was due to the limited model capacity. We contradict this hypothesis and show that a generalizable DR can be trained to achieve high accuracy in both supervised and zero-shot retrieval without increasing model size. In particular, we systematically examine the contrastive learning of DRs, under the framework of Da…
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers usin…
University of Southern California（南加州大学）；Tsinghua University（清华大学）
Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all. In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting. We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization. ReCross is a straightforward yet effective…
End-to-end speech recognition models are improved by incorporating external text sources, typically by fusion with an external language model. Such language models have to be retrained whenever the corpus of interest changes. Furthermore, since they store the entire corpus in their parameters, rare words can be challenging to recall. In this work, we propose augmenting a transducer-based ASR model with a retrieval language model, which directly retrieves from an external text corpus plausible completions for a partial ASR hypothesis. These completions are then integrated into subsequent predic…
