LongWeave:一座衔接现实相关性与可验证性的长文本生成基准
LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability
AI总结:
针对现有长文本生成基准难以兼顾现实性与可验证性的问题,该研究提出LongWeave基准及CoV-Eval评估方法,构建兼具真实性与可客观评估性的任务,实验表明先进LLM在长文本生成中仍面临显著挑战。
摘要:
生成篇幅长、信息量大且符合事实的输出仍是大语言模型(LLM)面临的一大挑战。现有的长文本生成基准通常采用难以验证的指标评估现实世界查询,或是使用合成设置——这类设置虽能简化评估,却忽略了现实世界的复杂细节。本文提出LongWeave基准,通过约束验证器评估法(Constraint-Verifier Evaluation, CoV-Eval)平衡现实性与可验证评估。CoV-Eval的任务构建流程为:先在现实场景中定义可验证目标,再基于这些目标系统生成对应的查询、文本材料与约束条件。这确保了任务既具现实性又可客观评估,能够严谨衡量模型满足复杂现实约束的能力。LongWeave支持7类不同任务,可自定义输入/输出长度(最高达64K/8K tokens)。对23个大语言模型的评估显示,随着现实复杂度提升与输出长度增加,即便最先进的模型在长文本生成中也会遭遇显著挑战。
英文摘要:
Generating long, informative, and factual outputs remains a major challenge for Large Language Models (LLMs). Existing benchmarks for long-form generation typically assess real-world queries with hard-to-verify metrics or use synthetic setups that ease evaluation but overlook real-world intricacies. In this paper, we introduce \textbf{LongWeave}, which balances real-world and verifiable assessment with Constraint-Verifier Evaluation (CoV-Eval). CoV-Eval constructs tasks by first defining verifiable targets within real-world scenarios, then systematically generating corresponding queries, textual materials, and constraints based on these targets. This ensures that tasks are both realistic and objectively assessable, enabling rigorous assessment of model capabilities in meeting complex real-world constraints. LongWeave supports customizable input/output lengths (up to 64K/8K tokens) across seven distinct tasks. Evaluation on 23 LLMs shows that even state-of-the-art models encounter significant challenges in long-form generation as real-world complexity and output length increase.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。