Despite the substantial success of Information Retrieval (IR) in various NLP tasks, most IR systems predominantly handle queries and corpora in natural language, neglecting the domain of code retrieval. Code retrieval is critically important yet remains under-explored, with existing methods and benchmarks inadequately representing the diversity of code in various domains and tasks. Addressing this gap, we present COIR (Code Information Retrieval Benchmark), a robust and comprehensive benchmark specifically designed to assess code retrieval capabilities. COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains. We first discuss the construction of COIR and its diverse dataset composition. Further, we evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems. To facilitate easy adoption and integration within existing research workflows, COIR has been developed as a user-friendly Python framework, readily installable via pip. It shares same data schema as other popular benchmarks like MTEB and BEIR, enabling seamless cross-benchmark evaluations. Through COIR, we aim to invigorate research in the code retrieval domain, providing a versatile benchmarking tool that encourages further development and exploration of code retrieval systems. https://github.com/CoIR-team/coir.
IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉科学学院）；Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究院）；Zhejiang University（浙江大学）；Yale University（耶鲁大学）
FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（中国科学院智能信息处理重点实验室）；University of Chinese Academy of Sciences, China（中国科学院大学）；Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）；Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen), China（哈尔滨工业大学（深圳）计算机与智能信息研究院）
Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；McGill University（麦吉尔大学）；Tencent Company（腾讯公司）；Artificial Intelligence Research Institute, iFLYTEK Co., Ltd（人工智能研究院，iFLYTEK公司）
PerSphere: A Comprehensive Framework for Multi-Faceted Perspective Retrieval and Summarization
School of Engineering, Westlake University（西湖大学工程学院）；Amazon AWS AI（亚马逊AWS人工智能部门）；Sichuan University（四川大学）；Shanghai AI Lab（上海人工智能实验室）；New York University Shanghai（上海纽约大学）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Computer Engineering Department, Sharif University of Technology（沙斐大学计算机工程系）；College of Interdisciplinary Science and Technology, University of Tehran（德黑兰大学跨学科科学与技术学院）；Computer Engineering Department, K.N. Toosi University of Technology（技术学院计算机工程系）；Qatar Computing Research Institute（卡塔尔计算研究院）
KnowRA: Knowledge Retrieval Augmented Method for Document-level Relation Extraction with Comprehensive Reasoning Abilities
Nanjing Normal University, School of Computer Science and Electronic Informatics（南京师范大学计算机科学与电子信息学院）；State Key Laboratory for Novel Software Technology at Nanjing University（南京大学新型软件技术国家重点实验室）；Nanjing University, School of Computer Science（南京大学计算机科学学院）
CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation
Chongqing University（重庆大学）；Independent Researcher（独立研究者）；University of the Chinese Academy of Sciences（中国科学院大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）
RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answering
Courant Institute School of Mathematics, Computing, and Data Science, New York University（纽约大学Courant研究所数学、计算与数据科学学院）；Courant Institute School of Mathematics, Computing（纽约大学Courant研究所数学、计算与数据科学学院）；Data Science, New York University（纽约大学数据科学学院）
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
University of Science and Technology of China（中国科学技术大学）；Institute for Advanced Algorithms Research (Shanghai)（（上海）高级算法研究院）；Renmin University of China（中国人民大学）；AI Research Institute（360人工智能研究院）
A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
MIRB: Mathematical Information Retrieval Benchmark
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Beijing International Center for Mathematical Research and the New Cornerstone Science Laboratory, Peking University（北京国际数学研究中心和新基石科学实验室，北京大学）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；Center for Intelligent Computing, Great Bay Institute for Advanced Study, Great Bay University（智能计算中心，大湾研究院高级研究所，大湾大学）
A Comparative Study of Specialized LLMs as Dense Retrievers
Key Laboratory of Network Data Science and Technology（网络数据科学与技术重点实验室）；Institute of Computing Technology（计算技术研究所）；Chinese Academy of Sciences（中国科学院）；State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；University of Chinese Academy of Science（中国科学院大学）
Evaluating LLM Abilities to Understand Tabular Electronic Health Records: A Comprehensive Study of Patient Data Extraction and Retrieval
Université Paul Sabatier（保罗·萨巴蒂埃大学）；IRIT（信息论与电信研究所）；Centre Hospitalier Universitaire de Toulouse（图卢兹大学医院中心）；INSERM（法国国家健康与医学研究院）；Université de Toulouse（图卢兹大学）；Faculté de Médecine（医学院）
CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院通用人工智能全国重点实验室）；SKLCCSE, School of Computer Science and Engineering, Beihang University, China（北京航空航天大学计算机科学与工程学院）
