Recent breakthroughs in Large Language Models (LLMs) have revolutionized scientific literature analysis. However, existing benchmarks fail to adequately evaluate the proficiency of LLMs in this domain, particularly in scenarios requiring higher-level abilities beyond mere memorization and the handling of multimodal data. In response to this gap, we introduce SciAssess, a benchmark specifically designed for the comprehensive evaluation of LLMs in scientific literature analysis. It aims to thoroughly assess the efficacy of LLMs by evaluating their capabilities in Memorization (L1), Comprehension (L2), and Analysis \& Reasoning (L3). It encompasses a variety of tasks drawn from diverse scientific fields, including biology, chemistry, material, and medicine. To ensure the reliability of SciAssess, rigorous quality control measures have been implemented, ensuring accuracy, anonymization, and compliance with copyright standards. SciAssess evaluates 11 LLMs, highlighting their strengths and areas for improvement. We hope this evaluation supports the ongoing development of LLM applications in scientific literature analysis. SciAssess and its resources are available at \url{https://github.com/sci-assess/SciAssess}.
PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University（香港理工大学）；University of Electronic Science and Technology of China（电子科技大学）；Tencent Youtu Lab（腾讯优图实验室）；Beihang University（北京航空航天大学）；Simon Fraser University（西蒙弗雷泽大学）
FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Shanghai Institute of AI for Education, East China Normal University（华东师范大学上海人工智能教育研究院）；PICC AMC（中国人民保险资产管理公司）；Bytedance（字节跳动）
MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts
Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Nankai University（南开大学）；Beijing Institute of Technology（北京理工大学）；Baichuan Inc.（百度文心）
Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking
Institute of Medical Education, Health Science Center, Peking University（北京大学医学教育研究院、健康科学中心）；School of Public Health, Peking University（北京大学公共卫生学院）；Peking University Health Science Center-Chaoxing Joint Laboratory for Digital and Smart Medical（北京大学健康科学中心-超星数字与智能医疗联合实验室）；Graduate School of Education, Peking University（北京大学教育学院）
Benchmarking Motivational Interviewing Competence of Large Language Models
National Institute of Mental Health and Neuro Sciences (NIMHANS)（国家精神卫生与神经科学研究所（NIMHANS））；Indian Institute of Technology (IIT) Kharagpur（印度理工学院卡拉格普尔分校（IIT Kharagpur））
EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning
Tencent AI Lab（腾讯 AI 实验室）；The University of Hong Kong（香港大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群 ARC 实验室）；University of California, Berkeley（加州大学伯克利分校）；Peng Cheng Laboratory（鹏城实验室）
MatTools: Benchmarking Large Language Models for Materials Science Tools
Center for Structural Materials, Department of Mechanical Engineering, The University of Hong Kong（香港大学机械工程系结构材料中心）；Materials Innovation Institute for Life Sciences（生命科学创新研究所）
Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncology
University Hospital Erlangen（埃尔兰根大学医院）；Comprehensive Cancer Center Erlangen-EMN（埃尔兰根-EMN综合癌症中心）；Pattern Recognition Lab（模式识别实验室）
