Large Language Models have evolved from single-round generators into long-horizon agents, capable of complex text synthesis scenarios. However, current evaluation frameworks lack the ability to assess the actual synthesis operations, such as outlining, drafting, and editing. Consequently, they fail to evaluate the actual and detailed capabilities of LLMs. To bridge this gap, we introduce RAVEL, an agentic framework that enables the LLM testers to autonomously plan and execute typical synthesis operations, including outlining, drafting, reviewing, and refining. Complementing this framework, we present C3EBench, a comprehensive benchmark comprising 1,258 samples derived from professional human writings. We utilize a "reverse-engineering" pipeline to isolate specific capabilities across four tasks: Cloze, Edit, Expand, and End-to-End. Through our analysis of 14 LLMs, we uncover that most LLMs struggle with tasks that demand contextual understanding under limited or under-specified instructions. By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability rather than raw generative capacity. Furthermore, we find that a strong reasoner can guide a weaker generator to yield higher-quality results, whereas the inverse does not hold. Our code and data are available at this link: https://github.com/ZhuoerFeng/RAVEL-Reasoning-Agents-Text-Eval.
The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models
LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Cambridge(剑桥大学);University of Maryland, College Park(马里兰大学帕克分校);ITU Copenhagen(哥本哈根信息技术大学)
CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
Agency for Science, Technology and Research (A*STAR)(科技研究局);Nanyang Technological University(南洋理工大学);Hong Kong University of Science and Technology (Guangzhou), China(香港科技大学(广州),中国)
OpenLearnLM Benchmark: A Unified Framework for Evaluating Knowledge, Skill, and Attitude in Educational Large Language Models
Chosun University(chosun大学);Korea University(韩国大学);Ewha Womans University(成均馆大学);Korea Institute for Curriculum and Evaluation(韩国课程评价院);Seoul National University(首尔国立大学);Texas A&M University(德克萨斯农工大学);Indiana University Bloomington(印第安纳大学布卢明顿分校)
EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI
Technical University of Munich(慕尼黑技术大学);MaiNLP, Center for Information and Language Processing, LMU Munich(MaiNLP,信息与语言处理中心,慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML))
AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment
Department of Computer Science, Sapienza University of Rome(计算机科学系,罗马萨皮恩扎大学);Department of Computer, Control, and Management Engineering, Sapienza University of Rome(计算机、控制与管理工程系,罗马萨皮恩扎大学);eZecute S.R.L.(eZecute公司)