长上下文语言模型能否取代检索、RAG、SQL及更多功能？
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
- Google DeepMind（谷歌DeepMind）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究推出百万token级上下文基准LOFT，评估发现长上下文语言模型虽未专门训练，仍可匹敌先进检索与RAG系统，但在组合推理类SQL任务上存在短板，提示策略影响显著，为相关范式革新提供测试基础。
AI中文摘要：
长上下文语言模型（LCLMs）有潜力革新我们处理传统依赖检索系统或数据库等外部工具的任务的方式。利用LCLMs原生读取和处理整个信息语料库的能力具备诸多优势：它无需使用者掌握工具的专业知识，提升了易用性；提供稳健的端到端建模，可减少复杂流水线中的级联错误；还能在整个系统中应用复杂的提示工程技术。为评估这一范式转变，我们推出LOFT（一个包含需百万级token上下文的真实世界任务的基准测试集），用于评估LCLMs在上下文内检索和推理方面的表现。研究发现，尽管LCLMs从未针对这些任务进行过显式训练，但其表现竟能与最先进的检索系统和RAG系统相媲美。不过，LCLMs在类SQL任务所需的组合推理等领域仍面临挑战。值得注意的是，提示策略对性能影响显著，这凸显出随着上下文长度增长，仍需持续开展相关研究。总体而言，LOFT为LCLMs提供了严格的测试平台，展现出随着模型能力提升，它们有望取代现有范式并解决新型任务。
英文摘要：
Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging LCLMs' ability to natively ingest and process entire corpora of information offers numerous advantages. It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system. To assess this paradigm shift, we introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks. However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks. Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow. Overall, LOFT provides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.
