This paper democratizes neural information retrieval to scenarios where large scale relevance training signals are not available. We revisit the classic IR intuition that anchor-document relations approximate query-document relevance and propose a reinforcement weak supervision selection method, ReInfoSelect, which learns to select anchor-document pairs that best weakly supervise the neural ranker (action), using the ranking performance on a handful of relevance labels as the reward. Iteratively, for a batch of anchor-document pairs, ReInfoSelect back propagates the gradients through the neural ranker, gathers its NDCG reward, and optimizes the data selection network using policy gradients, until the neural ranker's performance peaks on target relevance metrics (convergence). In our experiments on three TREC benchmarks, neural rankers trained by ReInfoSelect, with only publicly available anchor data, significantly outperform feature-based learning to rank methods and match the effectiveness of neural rankers trained with private commercial search logs. Our analyses show that ReInfoSelect effectively selects weak supervision signals based on the stage of the neural ranker training, and intuitively picks anchor-document pairs similar to query-document pairs.
Neural network approaches have recently shown to be effective in several information retrieval (IR) tasks. However, neural approaches often require large volumes of training data to perform effectively, which is not always available. To mitigate the shortage of labeled data, training neural IR models with weak supervision has been recently proposed and received considerable attention in the literature. In weak supervision, an existing model automatically generates labels for a large set of unlabeled data, and a machine learning model is further trained on the generated "weak" data. Surprisingl…
Although neural information retrieval has witnessed great improvements, recent works showed that the generalization ability of dense retrieval models on target domains with different distributions is limited, which contrasts with the results obtained with interaction-based models. To address this issue, researchers have resorted to adversarial learning and query generation approaches; both approaches nevertheless resulted in limited improvements. In this paper, we propose to use a self-supervision approach in which pseudo-relevance labels are automatically generated on the target domain. To do…
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland（马里兰大学）；Emory University（埃默里大学）；University of Pennsylvania（宾夕法尼亚大学）
Pretrained language models have improved effectiveness on numerous tasks, including ad-hoc retrieval. Recent work has shown that continuing to pretrain a language model with auxiliary objectives before fine-tuning on the retrieval task can further improve retrieval effectiveness. Unlike monolingual retrieval, designing an appropriate auxiliary task for cross-language mappings is challenging. To address this challenge, we use comparable Wikipedia articles in different languages to further pretrain off-the-shelf multilingual pretrained models before fine-tuning on the retrieval task. We show tha…
UIUC（伊利诺伊大学厄巴纳-香槟分校）；IBM Research AI（IBM研究院人工智能部门）
Neural passage retrieval is a new and promising approach in open retrieval question answering. In this work, we stress-test the Dense Passage Retriever (DPR) -- a state-of-the-art (SOTA) open domain neural retrieval model -- on closed and specialized target domains such as COVID-19, and find that it lags behind standard BM25 in this important real-world setting. To make DPR more robust under domain shift, we explore its fine-tuning with synthetic training examples, which we generate from unlabeled target domain text using a text-to-text generator. In our experiments, this noisy but fully autom…
UiT The Arctic University of Norway（挪威北极大学）；Indian Institute of Technology (Indian School of Mines), Dhanbad（印度理工学院（印度矿业学院）丹巴德分校）
Image retrieval has garnered growing interest in recent times. The current approaches are either supervised or self-supervised. These methods do not exploit the benefits of hybrid learning using both supervision and self-supervision. We present a novel Master Assistant Buddy Network (MABNet) for image retrieval which incorporates both learning mechanisms. MABNet consists of master and assistant blocks, both learning independently through supervision and collectively via self-supervision. The master guides the assistant by providing its knowledge base as a reference for self-supervision and the…
Towards Efficient Quantity Retrieval from Text:An Approach via Description Parsing and Weak Supervision
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS)（中国科学院智能信息处理重点实验室（中国科学院））；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences, CAS（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Ant Financial Services Group（蚂蚁集团）
Recent studies on Question Answering (QA) and Conversational QA (ConvQA) emphasize the role of retrieval: a system first retrieves evidence from a large collection and then extracts answers. This open-retrieval ConvQA setting typically assumes that each question is answerable by a single span of text within a particular passage (a span answer). The supervision signal is thus derived from whether or not the system can recover an exact match of this ground-truth answer span from the retrieved passages. This method is referred to as span-match weak supervision. However, information-seeking conver…
The limited availability of ground truth relevance labels has been a major impediment to the application of supervised methods to ad-hoc retrieval. As a result, unsupervised scoring methods, such as BM25, remain strong competitors to deep learning techniques which have brought on dramatic improvements in other domains, such as computer vision and natural language processing. Recent works have shown that it is possible to take advantage of the performance of these unsupervised methods to generate training data for learning-to-rank models. The key limitation to this line of work is the size of t…
Document retrieval is a core component of many knowledge-intensive natural language processing task formulations such as fact verification and question answering. Sources of textual knowledge, such as Wikipedia articles, condition the generation of answers from the models. Recent advances in retrieval use sequence-to-sequence models to incrementally predict the title of the appropriate Wikipedia page given a query. However, this method requires supervision in the form of human annotation to label which Wikipedia pages contain appropriate context. This paper introduces a distant-supervision met…
Co-adaptive Rule Routing and Constraining for Structured Knowledge Retrieval
University of Georgia（佐治亚大学）；Michigan State University（密歇根州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Amazon（亚马逊）；Adobe Systems（奥多比系统公司）；PNNL（太平洋西北国家实验室）；Texas A&M University（德克萨斯农工大学）
Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering
Faculty of Computing and Information Technology, King Abdulaziz University（计算机与信息科技学院，国王阿卜杜勒阿齐兹大学）；College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University (IMSIU)（计算机与信息科学学院，伊玛目穆罕默德·本·萨乌德伊斯兰大学）；School of Computing, Macquarie University（计算学院，麦考瑞大学）
Neural network representation learning frameworks have recently shown to be highly effective at a wide range of tasks ranging from radiography interpretation via data-driven diagnostics to clinical decision support. This often superior performance comes at the price of dramatically increased training data requirements that cannot be satisfied in every given institution or scenario. As a means of countering such data sparsity effects, distant supervision alleviates the need for scarce in-domain data by relying on a related, resource-rich, task for training. This study presents an end-to-end neu…
User intent classification is an important task in information retrieval. In this work, we introduce a revised taxonomy of user intent. We take the widely used differentiation between navigational, transactional and informational queries as a starting point, and identify three different sub-classes for the informational queries: instrumental, factual and abstain. The resulting classification of user queries is more fine-grained, reaches a high level of consistency between annotators, and can serve as the basis for an effective automatic classification process. The newly introduced categories h…
University of Amsterdam（阿姆斯特丹大学）；Ahold Delhaize（皇家阿霍德德尔海兹集团）
In this work we focus on multi-turn passage retrieval as a crucial component of conversational search. One of the key challenges in multi-turn passage retrieval comes from the fact that the current turn query is often underspecified due to zero anaphora, topic change, or topic return. Context from the conversational history can be used to arrive at a better expression of the current turn query, defined as the task of query resolution. In this paper, we model the query resolution task as a binary term classification problem: for each term appearing in the previous turns of the conversation deci…
Systems for Open-Domain Question Answering (OpenQA) generally depend on a retriever for finding candidate passages in a large corpus and a reader for extracting answers from those passages. In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages. We argue that this modeling choice is insufficiently expressive for dealing with the complexity of natural language questions. To address this, we define ColBERT-QA, which adapts the scalable neural retrieval model ColBERT to OpenQA. ColBERT creates fine-grained interactions b…
University of Montreal（蒙特利尔大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Renmin University of China（中国人民大学）；Beihang University（北京航空航天大学）；University of Copenhagen（哥本哈根大学）；Beijing Jiaotong University（北京交通大学）
In contrast to traditional exhaustive search, selective search first clusters documents into several groups before all the documents are searched exhaustively by a query, to limit the search executed within one group or only a few groups. Selective search is designed to reduce the latency and computation in modern large-scale search systems. In this study, we propose MICO, a Mutual Information CO-training framework for selective search with minimal supervision using the search logs. After training, MICO does not only cluster the documents, but also routes unseen queries to the relevant cluster…
Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our "recurring span retrieval" approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseu…
Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing
Aix Marseille Université（艾克斯-马赛大学）；Université de Toulon（土伦大学）；CNRS（法国国家科学研究中心）；IMT（法国国立高等电信学院）；Univ. de Toulouse（图卢兹大学）；Université de Toulouse（图卢兹大学）；INSPE（国家教育与教学高级学院）；UT2J（图卢兹第二让·饶勒斯大学）；Université Toulouse Capitole（图卢兹第一大学）；UT1（图卢兹第一大学）；Edinburgh Napier University（爱丁堡龙比亚大学）
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；CAS Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所人工智能安全重点实验室）
BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation
Université de Montéal（蒙特阿尔大学）；Mila - Quebec AI Institute（魁北克AI研究院）；McGill University（麦吉尔大学）；Apple MLR（苹果机器学习研究部）；IVADO, Canada（加拿大IVADO研究所）；CIFAR AI Chair（CIFAR人工智能主席）；ServiceNow Research（ServiceNow研究）
