arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.16191cs.CL
收藏

HelloBench:评估大型语言模型的长文本生成能力

HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

Beihang University(北京航空航天大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University(北京大学);M-A-P;University of Chinese Academy of Sciences(中国科学院大学);Nanjing University(南京大学)
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Son… 展开作者

Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, Kai Chen

AI总结:

本文提出了HelloBench基准和HelloEval评估方法,用于全面评估大型语言模型的长文本生成能力,实验发现当前主流模型在生成超4000词文本时存在显著局限。

摘要:

近年来,大型语言模型(LLMs)在各种任务(例如长上下文理解)中展现出了卓越的能力,并且已经提出了许多基准。然而,我们观察到长文本生成能力尚未得到充分研究。因此,我们引入了分层长文本生成基准(HelloBench),这是一个全面、真实场景和开放式的基准,用于评估LLMs在生成长文本方面的表现。基于Bloom分类法,HelloBench将长文本生成任务分为五个子任务:开放式QA、摘要、聊天、文本补全和启发式文本生成。此外,我们提出了分层长文本评估(HelloEval),这是一种与人类对齐的评估方法,可显著减少人工评估所需的时间和精力,同时与人工评估保持高度相关性。我们对大约30个主流LLMs进行了广泛实验,发现当前的LLMs缺乏长文本生成能力。具体而言,首先,无论指令是否包含显式或隐式的长度约束,我们观察到大多数LLMs无法生成超过4000词的文本。其次,我们观察到尽管一些LLMs能够生成更长的文本,但存在许多问题(例如严重的重复和质量下降)。第三,为了证明HelloEval的有效性,我们将其与传统指标(例如ROUGE、BLEU等)和LLM-as-a-Judge方法进行比较,结果显示HelloEval与人工评估的相关性最高。我们在https://github.com/Quehry/HelloBench发布了代码。

英文摘要:

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks (e.g., long-context understanding), and many benchmarks have been proposed. However, we observe that long text generation capabilities are not well investigated. Therefore, we introduce the Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs' performance in generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. Besides, we propose Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation method that significantly reduces the time and effort required for human evaluation while maintaining a high correlation with human evaluation. We have conducted extensive experiments across around 30 mainstream LLMs and observed that the current LLMs lack long text generation capabilities. Specifically, first, regardless of whether the instructions include explicit or implicit length constraints, we observe that most LLMs cannot generate text that is longer than 4000 words. Second, we observe that while some LLMs can generate longer text, many issues exist (e.g., severe repetition and quality degradation). Third, to demonstrate the effectiveness of HelloEval, we compare HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge methods, which show that HelloEval has the highest correlation with human evaluation. We release our code in https://github.com/Quehry/HelloBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑