arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.11802cs.CL
收藏

Counting-Stars:一个用于评估长上下文大语言模型的多证据、位置感知且可扩展的基准

Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Models

Tencent Hunyuan(腾讯混元)

Mingyang Song, Mao Zheng, Xuan Luo

AI总结:

本文提出Counting-Stars基准,通过搜索和推理两项计数任务评估长上下文LLM的多证据检索能力,实验发现Gemini 1.5 Pro总体最佳且GPT-4 Turbo最稳定,并指出模型在长文本和复杂任务上仍有改进空间。

摘要:

尽管近期在开发具有强大长上下文能力的大语言模型方面付出了努力,但长上下文基准的缺乏意味着对其性能了解甚少。为了缓解这一空白,本文提出了Counting-Stars,一个多证据、位置感知且可扩展的基准,旨在评估长上下文LLM的多证据检索能力。Counting-Stars包含两个基于计数的多个证据检索子任务:搜索和推理。使用Counting-Stars,我们进行了实验以评估几个长上下文LLM,包括GPT-4 Turbo、Gemini 1.5 Pro、Claude3 Opus、GLM-4和Moonshot-v1。大量实验结果表明,Gemini 1.5 Pro取得了最佳总体结果,而GPT-4 Turbo在各种任务中表现出最稳定的性能。此外,我们对这些已扩展以处理长上下文场景的LLM的分析表明,随着输入上下文长度和任务复杂度的增加,仍有很大的改进空间。

英文摘要:

Despite recent efforts to develop large language models with robust long-context capabilities, the lack of long-context benchmarks means that relatively little is known about their performance. To alleviate this gap, in this paper, we propose \textbf{Counting-Stars}, a multi-evidence, position-aware, and scalable benchmark designed to evaluate the multi-evidence retrieval capabilities of long-context LLMs. \textbf{Counting-Stars} comprises two counting-based multiple pieces of evidence retrieval sub-tasks: searching and reasoning. Using Counting-Stars, we conduct experiments to evaluate several long-context LLMs, including GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4, and Moonshot-v1. Extensive experimental results demonstrate that Gemini 1.5 Pro achieves the best overall results, while GPT-4 Turbo exhibits the most stable performance across various tasks. Furthermore, our analysis of these LLMs, which have been extended to handle long-context scenarios, indicates that significant room for improvement remains as the length of the input context and the complexity of the tasks increase.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑