HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Beihang University(北京航空航天大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University(北京大学);M-A-P;University of Chinese Academy of Sciences(中国科学院大学);Nanjing University(南京大学)
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Son… 展开作者
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, Kai Chen
In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks (e.g., long-context understanding), and many benchmarks have been proposed. However, we observe that long text generation capabilities are not well investigated. Therefore, we introduce the Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs' performance in generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. Besides, we propose Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation method that significantly reduces the time and effort required for human evaluation while maintaining a high correlation with human evaluation. We have conducted extensive experiments across around 30 mainstream LLMs and observed that the current LLMs lack long text generation capabilities. Specifically, first, regardless of whether the instructions include explicit or implicit length constraints, we observe that most LLMs cannot generate text that is longer than 4000 words. Second, we observe that while some LLMs can generate longer text, many issues exist (e.g., severe repetition and quality degradation). Third, to demonstrate the effectiveness of HelloEval, we compare HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge methods, which show that HelloEval has the highest correlation with human evaluation. We release our code in https://github.com/Quehry/HelloBench.
LR^2Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Zhongguancun Academy(中关村学院);Wuhan AI Research(武汉人工智能研究院)
OralMLLM-Bench: Evaluating Cognitive Capabilities of Multimodal Large Language Models in Dental Practice
Beijing Stomatological Hospital(北京口腔医院);School of Stomatology, Capital Medical University(首都医科大学口腔医学院);University of Minnesota(明尼苏达大学);The Hong Kong Polytechnic University(香港理工大学)
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
School of Informatics, Xiamen University(厦门大学信息学院);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能学院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室(厦门大学),文化和旅游部)
A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks
DJCAD, University of Dundee, United Kingdom(邓迪大学DJCAD部门);ARG-tech, SSEN, University of Dundee, United Kingdom(邓迪大学);School of Computer Science, University of Birmingham, United Kingdom(伯明翰大学计算机科学学院)
Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding
Instituto Politécnico Nacional (IPN)(墨西哥国立理工学院);Wollo University(沃洛大学);Saarland University(萨尔大学);Bahir Dar University(巴赫达尔大学);University of Hamburg(汉堡大学)
FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysis
National University of Singapore(国立新加坡大学);Estates Pte Ltd(6Estates公司);Asian Institute of Digital Finance(亚洲数字金融研究所);Hefei University of Technology(合肥工业大学);University of Science and Technology of China(中国科学技术大学)
AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
Alibaba Group(阿里巴巴集团);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学上海市智能信息处理重点实验室)
Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Bank of China(中国银行);School of Engineering Science, University of Chinese Academy of Sciences(中国科学院大学工程科学学院);State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences(中国科学院力学研究所非线性力学重点实验室)
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University(哈杰泰佩大学);Aselsan Research(阿塞尔桑研究院);Koç University(科克大学);Amsterdam UMC, University of Amsterdam(阿姆斯特丹大学医学中心,阿姆斯特丹大学);Amsterdam Public Health(阿姆斯特丹公共卫生研究院)