CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；LightSpeed Studios, Tencent Inc.（腾讯公司光速工作室）；Sun Yat-Sen University（中山大学）
Developing Large Language Models (LLMs) with robust long-context capabilities has been the recent research focus, resulting in the emergence of long-context LLMs proficient in Chinese. However, the evaluation of these models remains underdeveloped due to a lack of benchmarks. To address this gap, we present CLongEval, a comprehensive Chinese benchmark for evaluating long-context LLMs. CLongEval is characterized by three key features: (1) Sufficient data volume, comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability, accommodating to models with context windows size from 1K to 100K; (3) High quality, with over 2,000 manually annotated question-answer pairs in addition to the automatically constructed labels. With CLongEval, we undertake a comprehensive assessment of 6 open-source long-context LLMs and 2 leading commercial counterparts that feature both long-context abilities and proficiency in Chinese. We also provide in-depth analysis based on the empirical results, trying to shed light on the critical capabilities that present challenges in long-context settings. The dataset, evaluation scripts, and model outputs are released.
Evaluating Chinese Ambiguity Understanding in Large Language Models
Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）；School of Software Engineering, South China University of Technology（华南理工大学软件学院）
CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns
Nanyang Technological University（南洋理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Huazhong University of Science and Technology（华中科技大学）
CHBench: A Chinese Dataset for Evaluating Health in Large Language Models
School of Artificial intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University（吉林大学知识驱动人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际中心）
Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry
College of Computer Science, Inner Mongolia University, China（内蒙古大学计算机学院，中国）；National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, China（蒙古语智能信息处理技术国家与地方联合工程研究中心，中国）；NLP²CT Lab, Department of Computer and Information Science, University of Macau, China（澳门大学计算机与信息科学学院NLP²CT实验室，中国）
CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Jilin University（吉林大学）；University of Science and Technology of China（中国科学技术大学）；Shenzhen University of Advanced Technology（深圳理工大学）；University of Macau（澳门大学）；Huazhong University of Science and Technology（华中科技大学）
Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Lab of Artificial Intelligence for Education, East China Normal University（华东师范大学教育人工智能实验室）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海教育人工智能研究院）
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University（华东师范大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Hasso Plattner Institute（哈索·普拉特纳研究所）；University of Potsdam（波茨坦大学）；Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作媒体网络创新中心）
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models
HKUST（香港科技大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；Ohio State University（俄亥俄州立大学）；AIWaves Inc.（AIWaves公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, Chinese Academy of Sciences（中国科学院人工智能学院）；Peking University（北京大学）；Beijing Foreign Studies University（北京外国语大学）；University of Manchester（曼彻斯特大学）；University of Waterloo（滑铁卢大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）
MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Institute of Infectious Disease and Biosecurity, Fudan University（复旦大学上海传染病与生物安全研究院）；Shanghai Health Development Research Center (Shanghai Medical Information Center)（上海市卫生和健康发展研究中心（上海市医学科学技术情报研究所））；Imperial College London（伦敦帝国理工学院）
Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers
Department of Electronics, Information and Bioengineering, Politecnico di Milano（电子、信息与生物工程系，米兰理工学院）；University of Science and Technology of China（中国科学技术大学）
Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?
Graduate School of Informatics, Kyoto University（京都大学信息学研究科）；University of Macau（澳门大学）；College of Computer Science, Inner Mongolia University（内蒙古大学计算机学院）
TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
Shanghai Institute of AI for Education, East China Normal University（华东师范大学人工智能教育研究院）；School of Pharmacy, East China Normal University（华东师范大学药学院）；Shanghai Skin Disease Hospital, School of Medicine, Tongji University（同济大学医学院附属上海市皮肤病医院）
MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
School of Software Engineering, Sun Yat-sen University（中山大学软件学院）；Institute of TCM Diagnostics, Hunan University of Chinese Medicine（湖南中医药大学中医诊断研究所）；School of Artificial Intelligence, Guilin University of Electronic Technology（桂林电子科技大学人工智能学院）；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）
