The principal goal of the TREC Neural Cross-Language Information Retrieval (NeuCLIR) track is to study the impact of neural approaches to cross-language information retrieval. The track has created four collections, large collections of Chinese, Persian, and Russian newswire and a smaller collection of Chinese scientific abstracts. The principal tasks are ranked retrieval of news in one of the three languages, using English topics. Results for a multilingual task, also with English topics but with documents from all three newswire collections, are also reported. New in this second year of the track is a pilot technical documents CLIR task for ranked retrieval of Chinese technical documents using English topics. A total of 220 runs across all tasks were submitted by six participating teams and, as baselines, by track coordinators. Task descriptions and results are presented.
This paper reports on a study of cross-lingual information retrieval (CLIR) using the mT5-XXL reranker on the NeuCLIR track of TREC 2022. Perhaps the biggest contribution of this study is the finding that despite the mT5 model being fine-tuned only on query-document pairs of the same language it proved to be viable for CLIR tasks, where query-document pairs are in different languages, even in the presence of suboptimal first-stage retrieval performance. The results of the study show outstanding performance across all tasks and languages, leading to a high number of winning positions. Finally, …
This paper describes our participation in the 2022 TREC NeuCLIR challenge. We submitted runs to two out of the three languages (Farsi and Russian), with a focus on first-stage rankers and comparing mono-lingual strategies to Adhoc ones. For monolingual runs, we start from pretraining models on the target language using MLM+FLOPS and then finetuning using the MSMARCO translated to the language either with ColBERT or SPLADE as the retrieval model. While for the Adhoc task, we test both query translation (to the target language) and back-translation of the documents (to English). Initial result a…
NeuCLIRTech: Chinese Monolingual and Cross-Language Information Retrieval Evaluation in a Challenging Domain
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Glasgow（格拉斯哥大学）；University of Waterloo（滑铁卢大学）；Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）；Allen Institute for AI（艾伦人工智能研究所）
We present an overview of the TREC-COVID Challenge, an information retrieval (IR) shared task to evaluate search on scientific literature related to COVID-19. The goals of TREC-COVID include the construction of a pandemic search test collection and the evaluation of IR methods for COVID-19. The challenge was conducted over five rounds from April to July, 2020, with participation from 92 unique teams and 556 individual submissions. A total of 50 topics (sets of related queries) were used in the evaluation, starting at 30 topics for Round 1 and adding 5 new topics per round to target emerging to…
Overview of the TREC 2025 Product Search and Recommendation Track
Department of Computer Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；Coursera（可汗学院）；Snowflake（雪佛龙）；Coupang；Waseda University（早稻田大学）；Department of Information Science, Drexel University（德雷塞尔大学信息科学系）
The TREC Fair Ranking Track aims to provide a platform for participants to develop and evaluate novel retrieval algorithms that can provide a fair exposure to a mixture of demographics or attributes, such as ethnicity, that are represented by relevant documents in response to a search query. For example, particular demographics or attributes can be represented by the documents topical content or authors. The 2022 Fair Ranking Track adopted a resource allocation task. The task focused on supporting Wikipedia editors who are looking to improve the encyclopedia's coverage of topics under the purv…
The TREC Fair Ranking Track aims to provide a platform for participants to develop and evaluate novel retrieval algorithms that can provide a fair exposure to a mixture of demographics or attributes, such as ethnicity, that are represented by relevant documents in response to a search query. For example, particular demographics or attributes can be represented by the documents' topical content or authors. The 2021 Fair Ranking Track adopted a resource allocation task. The task focused on supporting Wikipedia editors who are looking to improve the encyclopedia's coverage of topics under the pur…
Spotify（声田）；CLARIN ERIC；Dublin City University（都柏林城市大学）
The Podcast Track is new at the Text Retrieval Conference (TREC) in 2020. The podcast track was designed to encourage research into podcasts in the information retrieval and NLP research communities. The track consisted of two shared tasks: segment retrieval and summarization, both based on a dataset of over 100,000 podcast episodes (metadata, audio, and automatic transcripts) which was released concurrently with the track. The track generated considerable interest, attracted hundreds of new registrations to TREC and fifteen teams, mostly disjoint between search and summarization, made final s…
Johns Hopkins University Human Language Technology Center of Excellence（约翰霍普金斯大学人机语言技术卓越中心）；University of Glasgow（格拉斯哥大学）；Allen Institute for AI（人工智能研究院）
Microsoft AI & Research（微软人工智能研究院）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available, with much more comprehensive relevance labeling on the small number of test queries. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in th…
An overview on the evaluated video retrieval tasks at TRECVID 2022
National Institute of Standards and Technology（美国国家标准与技术研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Dakota-consulting（达科他咨询公司）；MIT Lincoln Laboratory（麻省理工学院林肯实验室）；Trinity College Dublin（都柏林圣三一学院）
Microsoft Research Montréal（微软研究院蒙特利尔分院）；Boise State University（博伊西州立大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different content providers in addition to classic notions of relevance. As part of the benchmark, we defined standardized fairness metrics with evaluation protocols and released a dataset for the fair ranking problem. The 2019 task focused on reranking academic paper abstracts given a query. The objective was to fairly represent relevant authors from several groups that were unknown at the system submission time. Thus, the track emphasized the development of systems whic…
Microsoft Research Montréal（微软研究院蒙特利尔）；Boise State University（博伊西州立大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）
This paper provides an overview of the NIST TREC 2020 Fair Ranking track. For 2020, we again adopted an academic search task, where we have a corpus of academic article abstracts and queries submitted to a production academic search engine. The central goal of the Fair Ranking track is to provide fair exposure to different groups of authors (a group fairness framing). We recognize that there may be multiple group definitions (e.g. based on demographics, stature, topic) and hoped for the systems to be robust to these. We expected participants to develop systems that optimize for fairness and re…
University of Glasgow（格拉斯哥大学）；Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
The Conversational Assistance Track (CAsT) is a new track for TREC 2019 to facilitate Conversational Information Seeking (CIS) research and to create a large-scale reusable test collection for conversational search systems. The document corpus is 38,426,252 passages from the TREC Complex Answer Retrieval (CAR) and Microsoft MAchine Reading COmprehension (MARCO) datasets. Eighty information seeking dialogues (30 train, 50 test) are an average of 9 to 10 questions long. Relevance assessments are provided for 30 training topics and 20 test topics. This year 21 groups submitted a total of 65 runs …
Microsoft AI & Research（微软人工智能与研究部）；University College London（伦敦大学学院）；NIST（美国国家标准与技术研究院（NIST））
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets. The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries. The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test se…
Retrieving paragraphs to populate a Wikipedia article is a challenging task. The new TREC Complex Answer Retrieval (TREC CAR) track introduces a comprehensive dataset that targets this retrieval scenario. We present early results from a variety of approaches -- from standard information retrieval methods (e.g., tf-idf) to complex systems that using query expansion using knowledge bases and deep neural networks. The goal is to offer future participants of this track an overview of some promising approaches to tackle this problem.
Johns Hopkins University（约翰霍普金斯大学）；University of New Hampshire（新罕布什尔大学）；IDA Center for Computing Services（IDA计算服务中心）；Google（谷歌）；Yale University（耶鲁大学）；University of Maryland（马里兰大学）
Microsoft（微软公司）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Neural Magic Inc（Neural Magic公司）；University of Waterloo（滑铁卢大学）
Microsoft（微软公司）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Neural Magic Inc（Neural Magic公司）；University of Waterloo（滑铁卢大学）；NIST（国家标准与技术研究院）
本文概述了 TREC 2022 深度学习赛道，利用 MS MARCO 数据集和扩充后的集合构建了更完整的段落检索测试集，并证实大规模预训练的深度神经排序模型仍优于传统方法。
Microsoft（微软）；University College London（伦敦大学学院）；University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；National Institute of Standards and Technology（美国国家标准与技术研究院）
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that the best models with large data may be deep neural networks. This paper supports the reuse of the TREC DL test collections in three ways. First we describe the data sets in detail, documenting clearly and in one place some details that are otherwise scattered in track guidelines, overview papers and in our associated MS MARCO leaderboard pages. We intend this description to make it easy for newcomers to use the TREC…
