Current benchmarks like Needle-in-a-Haystack (NIAH), Ruler, and Needlebench focus on models' ability to understand long-context input sequences but fail to capture a critical dimension: the generation of high-quality long-form text. Applications such as design proposals, technical documentation, and creative writing rely on coherent, instruction-following outputs over extended sequences - a challenge that existing benchmarks do not adequately address. To fill this gap, we introduce LongGenBench, a novel benchmark designed to rigorously evaluate large language models' (LLMs) ability to generate long text while adhering to complex instructions. Through tasks requiring specific events or constraints within generated text, LongGenBench evaluates model performance across four distinct scenarios, three instruction types, and two generation-lengths (16K and 32K tokens). Our evaluation of ten state-of-the-art LLMs reveals that, despite strong results on Ruler, all models struggled with long text generation on LongGenBench, particularly as text length increased. This suggests that current LLMs are not yet equipped to meet the demands of real-world, long-form text generation.
Benchmarking LLMs on the Semantic Overlap Summarization Task
Research Center Trustworth Data Science and Security, Technical University of Dortmund（多特蒙德技术大学信任数据科学与安全研究中心）；School of Computing, Wichita State University（威斯康星州立大学计算机学院）；Department of Computer Science, NC State University（北卡罗来纳州立大学计算机科学系）
Redefining Simplicity: Benchmarking Large Language Models from Lexical to Document Simplification
School of Information and Engineering, Yangzhou University（扬州大学信息工程学院）；School of Computer Science and Information Technology, Hefei University of Technology（合肥工业大学计算机与信息学院）
OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；School of Computer and Artificial Intelligence, Zhengzhou University（郑州大学计算机与人工智能学院）
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
The Chinese University of Hong Kong（香港中文大学）；Tianjin Medical University（天津医科大学）；Tencent AI Lab（腾讯AI实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios
TJUNLP Lab, School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院TJUNLP实验室）；The International Joint Institute of Tianjin University（天津大学国际联合研究院）
DependEval: Benchmarking LLMs for Repository Dependency Understanding
Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；ShanghaiTech University（上海科技大学）；The University of Hong Kong（香港大学）
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University（香港理工大学）；University of Electronic Science and Technology of China（电子科技大学）；Tencent Youtu Lab（腾讯优图实验室）；Beihang University（北京航空航天大学）；Simon Fraser University（西蒙弗雷泽大学）
CANDY: Benchmarking LLMs' Limitations and Assistive Potential in Chinese Misinformation Fact-Checking
School of Cyber Science and Engineering, Sichuan University, China（四川大学网络科学与工程学院）；College of Computer Science, Sichuan University, China（四川大学计算机学院）；Institute of Data Science, National University of Singapore, Singapore（新加坡国立大学数据科学研究所）
BioCoref: Benchmarking Biomedical Coreference Resolution with LLMs
Computational Bioscience Program University of Colorado Anschutz Medical Campus（科学生物学程序，科罗拉多大学安舒茨医学校区）；Department of Pediatrics University of Chicago（儿科学系，芝加哥大学）
Benchmarking Resource-Efficient LLMs for Research Topic Ontology Generation in the Biomedical Field
Knowledge Media Institute, The Open University, Milton Keynes, UK（开放大学知识媒体研究所）；Department of Business and Law, University of Milano-Bicocca, Milan, IT（米兰-比科卡大学商业与法律系）
