arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.00686cs.CL
收藏

RAVEL: 用于验证和评估大语言模型文本合成的推理代理

RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis

Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo, Bosi Wen, Yidong Wang, Lin Fan, Yilin Zhou, Zikang Wang, Wenbo Yu, Lindong Wu, Hongning Wang, Minlie Huang

AI总结:

RAVEL通过引入智能体框架和C3EBench基准,评估LLM在文本合成中的推理能力,发现推理能力比生成能力更重要。

摘要:

大型语言模型已经从单轮生成器发展为具有长周期能力的智能体,能够处理复杂的文本合成场景。然而,当前的评估框架缺乏评估实际合成操作(如大纲、草稿、编辑)的能力。因此,它们无法评估LLM的实际和详细能力。为弥合这一差距,我们引入了RAVEL,一个能够使LLM测试者自主规划和执行典型合成操作(包括大纲、草稿、审查和润色)的框架。此外,我们还提出了C3EBench,一个包含1258个样本的全面基准,这些样本来源于专业人类写作。我们使用

英文摘要:

Large Language Models have evolved from single-round generators into long-horizon agents, capable of complex text synthesis scenarios. However, current evaluation frameworks lack the ability to assess the actual synthesis operations, such as outlining, drafting, and editing. Consequently, they fail to evaluate the actual and detailed capabilities of LLMs. To bridge this gap, we introduce RAVEL, an agentic framework that enables the LLM testers to autonomously plan and execute typical synthesis operations, including outlining, drafting, reviewing, and refining. Complementing this framework, we present C3EBench, a comprehensive benchmark comprising 1,258 samples derived from professional human writings. We utilize a "reverse-engineering" pipeline to isolate specific capabilities across four tasks: Cloze, Edit, Expand, and End-to-End. Through our analysis of 14 LLMs, we uncover that most LLMs struggle with tasks that demand contextual understanding under limited or under-specified instructions. By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability rather than raw generative capacity. Furthermore, we find that a strong reasoner can guide a weaker generator to yield higher-quality results, whereas the inverse does not hold. Our code and data are available at this link: https://github.com/ZhuoerFeng/RAVEL-Reasoning-Agents-Text-Eval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑