Large language models (LLMs) garner significant attention for their unprecedented performance, leading to an increasing number of researches evaluating LLMs. However, these evaluation benchmarks are limited to assessing the instruction-following capabilities, overlooking the fundamental abilities that emerge during the pre-training stage. Previous subjective evaluation methods mainly reply on scoring by API models. However, in the absence of references, large models have shown limited ability to discern subtle differences. To bridge the gap, we propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic. The tasks in F-Eval include multi-choice objective tasks, open-ended objective tasks, reference-based subjective tasks and reference-free subjective tasks. For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models. We conduct evaluations on 13 advanced LLMs. Results show that our evaluation methods show higher correlation coefficients and larger distinction than other evaluators. Additionally, we discuss the influence of different model sizes, dimensions, and normalization methods. We anticipate that F-Eval will facilitate the study of LLMs' fundamental abilities.
AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
Alibaba Group（阿里巴巴集团）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学上海市智能信息处理重点实验室）
KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
Huawei Poisson Laboratory（华为泊松实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Hong Kong University of Science and Technology（香港科技大学）
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Data Science and Intelligent Media, Communication University of China（中国传媒大学数据科学与智能媒体学院）
ICLEval: Evaluating In-Context Learning Ability of Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Huawei Poisson Lab（华为泊松实验室）；School of Information, Renmin University of China（中国人民大学信息学院）
Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs
University of Leeds（利兹大学）；The Alan Turing Institute（艾伦·图灵研究所）；Tongji University（同济大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；ValGRAI；Leverhulme Centre for the Future of Intelligence（利弗休姆未来智能中心）
Predicting Emergent Abilities with Infinite Resolution Evaluation
Tsinghua University（清华大学）；Beijing Language and Culture University（北京语言大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Renmin University of China（中国人民大学）；Zhihu Inc.（知乎）；Modelbest Inc.（模贝斯特公司）
Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities
Sun Yat-sen University（中山大学）；Tsinghua University（清华大学）；ByteDance Inc（字节跳动公司）；Arizona State University（亚利桑那州立大学）；Tencent Youtu Lab（腾讯优图实验室）；Fudan University（复旦大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology（广东省消防科学与智能应急技术重点实验室）；University of Illinois Chicago（伊利诺伊大学香槟分校）
Pet-Bench: Benchmarking the Abilities of Large Language Models as E-Pets in Social Network Services
Fudan University（复旦大学）；Xiaohongshu Inc.（小红书公司）；Beihang University（北航）；Zhejiang University（浙江大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Xi’an Jiaotong-Liverpool University（西交利物浦大学）；University of Liverpool（利物浦大学）；University of Macau（澳门大学）；HKUST（香港科技大学）；Microsoft Research Asia（微软亚洲研究院）；Duke Kunshan University（昆山杜克大学）
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
Olin College of Engineering（奥林工程学院）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Oxford（牛津大学）；University College London（伦敦大学学院）；ShanghaiTech University（上海科技大学）；Columbia University（哥伦比亚大学）；University of Manchester（曼彻斯特大学）
Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need
Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院（BAAI））；China University of Mining and Technology Beijing（中国矿业大学（北京））；Peking University（北京大学）
Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Georgia Institute of Technology（佐治亚理工学院）；The Hong Kong University of Science and Technology（香港科技大学）
Evaluating LLM Abilities to Understand Tabular Electronic Health Records: A Comprehensive Study of Patient Data Extraction and Retrieval
Université Paul Sabatier（保罗·萨巴蒂埃大学）；IRIT（信息论与电信研究所）；Centre Hospitalier Universitaire de Toulouse（图卢兹大学医院中心）；INSERM（法国国家健康与医学研究院）；Université de Toulouse（图卢兹大学）；Faculté de Médecine（医学院）
Can multiple-choice questions really be useful in detecting the abilities of LLMs?
Meetyou AI Lab（MeetYou人工智能实验室）；Southwestern University of Finance and Economics（西南财经大学）；Osaka University（大阪大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation
Department of Electrical and Computer Engineering, University of Pittsburgh（匹兹堡大学电气与计算机工程系）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Department of Computer Science, University of Texas Rio Grande Valley（德克萨斯理工大学里奥格兰德谷分校计算机科学系）
FRAME: Feedback-Refined Agent Methodology for Enhancing Medical Research Insights
South China University of Technology（南方科技大学）；HFIPS, Chinese Academy of Sciences（中国科学院HFIPS）；University of Science and Technology of China（中国科学技术大学）；The Third Affiliated Hospital of SuYat-sen University（孙逸德大学第三附属医院）
Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；BNRist, Tsinghua University（清华大学BNRist）
