arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12640cs.CLcs.LG
收藏

Michelangelo:通过潜在结构查询超越干草堆的长上下文评估

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院)
Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Roh… 展开作者

Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazaridou, Jean-Baptiste Lespiau, Nithya Attaluri, Kate Olszewska

AI总结:

本文提出了Michelangelo长上下文推理评估框架,通过潜在结构查询(LSQ)要求模型剔除无关信息以揭示潜在结构,并在代码和自然语言领域生成三项诊断性评估,验证了其高信号量并揭示了现有模型在长上下文信息综合方面的改进空间。

摘要:

我们介绍了 Michelangelo:一种针对大型语言模型的最小化、合成且未泄露的长上下文推理评估,同时也易于自动评分。该评估通过一种新颖且统一的框架构建,适用于任意长度的上下文,旨在衡量模型从其上下文中执行超越检索单一信息的能力。潜在结构查询框架(LSQ)的核心思想是构建任务,要求模型“剔除”上下文中的无关信息,从而揭示上下文中的潜在结构。为验证模型对该潜在结构的理解,我们向模型查询该结构的细节。利用 LSQ,我们在代码和自然语言领域产出了三项诊断性长上下文评估,旨在为长上下文语言模型能力提供更强的信号。我们在多个最先进的模型上进行了评估,并证明:a) 所提出的评估具有高信号量;b) 在合成(综合)长上下文信息方面仍有显著的改进空间。

英文摘要:

We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation is derived via a novel, unifying framework for evaluations over arbitrarily long contexts which measure the model's ability to do more than retrieve a single piece of information from its context. The central idea of the Latent Structure Queries framework (LSQ) is to construct tasks which require a model to ``chisel away'' the irrelevant information in the context, revealing a latent structure in the context. To verify a model's understanding of this latent structure, we query the model for details of the structure. Using LSQ, we produce three diagnostic long-context evaluations across code and natural-language domains intended to provide a stronger signal of long-context language model capabilities. We perform evaluations on several state-of-the-art models and demonstrate both that a) the proposed evaluations are high-signal and b) that there is significant room for improvement in synthesizing long-context information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑