LLMs and RAG systems are now capable of handling millions of input tokens or more. However, evaluating the output quality of such systems on long-context tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity. In this work, we argue that summarization can play a central role in such evaluation. We design a procedure to synthesize Haystacks of documents, ensuring that specific \textit{insights} repeat across documents. The "Summary of a Haystack" (SummHay) task then requires a system to process the Haystack and generate, given a query, a summary that identifies the relevant insights and precisely cites the source documents. Since we have precise knowledge of what insights should appear in a haystack summary and what documents should be cited, we implement a highly reproducible automatic evaluation that can score summaries on two aspects - Coverage and Citation. We generate Haystacks in two domains (conversation, news), and perform a large-scale evaluation of 10 LLMs and corresponding 50 RAG systems. Our findings indicate that SummHay is an open challenge for current systems, as even systems provided with an Oracle signal of document relevance lag our estimate of human performance (56\%) by 10+ points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to study enterprise RAG systems and position bias in long-context models. We hope future systems can equal and surpass human performance on SummHay.
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);College of Design and Innovation, Tongji University(同济大学设计与创新学院)
Adversarial Databases Improve Success in Retrieval-based Large Language Models
Keck Data Science Institute, Pepperdine University(佩珀代因大学凯克数据科学研究所);David Geffen School of Medicine, University of California, Los Angeles(加州大学洛杉矶分校大卫·格芬医学院)
Biomedical knowledge graph-optimized prompt generation for large language models
University of California San Francisco(加州大学旧金山分校);San Diego Supercomputer Center, University of California, San Diego(加州大学圣地亚哥分校圣地亚哥超级计算机中心);University of California, San Francisco(加州大学旧金山分校);Institute for Systems Biology(系统生物学研究所);Bakar Computational Health Sciences Institute, University of California, San Francisco(加州大学旧金山分校巴卡尔计算健康科学研究所);Mate Bioservices, Inc.(Mate 生物服务公司)
Towards Reliable Retrieval in RAG Systems for Large Legal Datasets
Technical University of Darmstadt(德意志联邦共和国达姆斯塔特技术大学);University of Trento(特伦托大学);University of Florence(佛罗伦萨大学);University of Bologna(博洛尼亚大学);European University Institute(欧洲大学研究所)
Self-Describing Structured Data with Dual-Layer Guidance: A Lightweight Alternative to RAG for Precision Retrieval in Large-Scale LLM Knowledge Navigation