HELMET:如何有效且全面地评估长上下文语言模型
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
AI总结:
提出综合基准HELMET,通过七类应用任务、可控长度和可靠指标评估长上下文模型,发现NIAH等合成任务无法预测下游性能,并揭示开闭源模型在复杂推理上的差距随长度扩大。
摘要:
许多用于评估长上下文语言模型(LCLMs)的基准已经存在,然而开发者们通常依赖于诸如大海捞针(NIAH)等合成任务或任意选取的任务子集。但是,这些基准是否能反映LCLMs多样化的下游应用仍不明确,且这种不一致性进一步使模型比较变得复杂。我们调查了这些做法背后的深层原因,发现现有基准由于应用覆盖范围有限、上下文长度不足、指标不可靠以及与基础模型不兼容,常常提供带有噪声的信号。在本研究中,我们提出了HELMET(How to Evaluate Long-context Models Effectively and Thoroughly,如何有效且全面地评估长上下文模型),这是一个包含七个多样化、以应用为中心类别的综合基准。我们还通过添加最多128K tokens的可控长度、采用基于模型的评估以提供可靠指标,以及使用少样本提示来稳健评估基础模型,解决了先前基准中的若干问题。因此,我们证明HELMET为前沿LCLMs提供了更可靠和一致的排名。通过对59个LCLMs的全面研究,我们发现:(1)像NIAH这样的合成任务无法可靠地预测下游性能;(2)HELMET中的多样化类别表现出独特的趋势,且彼此之间相关性较低;(3)尽管大多数LCLMs在NIAH上获得满分,但当任务需要全上下文推理或遵循复杂指令时,开源模型显著落后于闭源模型——且随着长度增加,这一差距不断扩大。最后,我们建议使用我们的RAG任务进行快速模型开发,因为它们易于运行且能更好地预测其他下游性能;最终,我们提倡在多样化任务中进行整体评估。
英文摘要:
Many benchmarks exist for evaluating long-context language models (LCLMs), yet developers often rely on synthetic tasks such as needle-in-a-haystack (NIAH) or an arbitrary subset of tasks. However, it remains unclear whether these benchmarks reflect the diverse downstream applications of LCLMs, and such inconsistencies further complicate model comparison. We investigate the underlying reasons behind these practices and find that existing benchmarks often provide noisy signals due to limited coverage of applications, insufficient context lengths, unreliable metrics, and incompatibility with base models. In this work, we introduce HELMET (How to Evaluate Long-context Models Effectively and Thoroughly), a comprehensive benchmark encompassing seven diverse, application-centric categories. We also address several issues in previous benchmarks by adding controllable lengths up to 128K tokens, model-based evaluation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, we demonstrate that HELMET offers more reliable and consistent rankings of frontier LCLMs. Through a comprehensive study of 59 LCLMs, we find that (1) synthetic tasks like NIAH do not reliably predict downstream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlations with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when tasks require full-context reasoning or following complex instructions -- the gap widens as length increases. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and better predict other downstream performance; ultimately, we advocate for a holistic evaluation across diverse tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。