BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
The University of Hong Kong(香港大学);Princeton University(普林斯顿大学);Stanford University(斯坦福大学);University of Washington(华盛顿大学);Google Cloud AI Research(谷歌云人工智能研究)
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon… 展开作者
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, Tao Yu
Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.
TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);Hong Kong Polytechnic University(香港理工大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval
University of Science and Technology of China(中国科学技术大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beijing University of Posts and Telecommunications(北京邮电大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Hong Kong Polytechnic University(香港理工大学)
Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval
University of Science and Technology of China(中国科学技术大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beijing University of Posts and Telecommunications(北京邮电大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Hong Kong Polytechnic University(香港理工大学);Hong Kong University of Science and Technology(香港科技大学)
Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学)
Allen Institute for AI(艾伦人工智能研究所);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Southern California(南加州大学);University of Washington(华盛顿大学)
本文提出网络规模高质量数据存储 CompactDS,结合内存 ANN 检索与磁盘精确搜索,使最小化 RAG 在多个推理密集型基准上一致提升准确率,并可与搜索引擎及复杂智能体 RAG 系统相媲美。
On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains
Department of ECE University of Minnesota(电子工程系明尼苏达大学);Data Science Institute University of Chicago(数据科学研究所芝加哥大学);Carlson School of Management University of Minnesota(卡尔森管理学院明尼苏达大学);Department of Surgery University of Minnesota(外科系明尼苏达大学);Cisco Research(思科研究);School of Statistics University of Minnesota(统计学学院明尼苏达大学)
CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of Amsterdam(阿姆斯特丹大学)
ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion
City University of Hong Kong(香港城市大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);MILA, McGill University(麦吉尔大学MILA研究所);National Taiwan University(国立台湾大学)
MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering
Nanyang Technological University(南洋理工大学);Alibaba-NTU JRI(阿里巴巴-南洋理工大学联合研究院);NYU Shanghai(上海纽约大学);Zhejiang University(浙江大学);Center for Data Science, New York University(纽约大学数据科学中心)
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer
Carnegie Mellon University(卡内基梅隆大学);Bosch Research North America(博世北美研究院);Amazon(亚马逊)
知识密集型任务的系统,如开放领域的问答(QA)通常包括两个阶段:从一个大的语料库中有效检索相关的文件和详细阅读所选择的文件,以产生答案。检索器和读取器通常是单独建模的,这需要繁琐的实现,并且很难以端到端的方式进行训练和调整。在本文中,我们重新审视了这种设计,并避开了单独的架构和培训,而是使用单个Transformer来执行Retrieval as Attention(ReAtt),并仅基于来自最终QA任务的监督进行端到端培训。我们第一次证明,一个单一的模型训练的端到端可以实现竞争检索和QA性能,匹配或略优于最先进的分别训练检索和读者。此外,端到端自适应显著提高了其在有监督和无监督环境下对域外数据集的性能,使我们的模型成为知识密集型任务的简单且适应性强的解决方案。代码和模型可在https://github.com/jzbjyb/ReAtt上获得。