Evaluating the graph comprehension and reasoning abilities of Large Language Models (LLMs) is challenging and often incomplete. Existing benchmarks focus primarily on pure graph understanding, lacking a comprehensive evaluation across all graph types and detailed capability definitions. This paper presents GraCoRe, a benchmark for systematically assessing LLMs' graph comprehension and reasoning. GraCoRe uses a three-tier hierarchical taxonomy to categorize and test models on pure graph and heterogeneous graphs, subdividing capabilities into 10 distinct areas tested through 19 tasks. Our benchmark includes 11 datasets with 5,140 graphs of varying complexity. We evaluate four closed-source and eight open-source LLMs, conducting thorough analyses from both ability and task perspectives. Key findings reveal that OpenAI o1 model has amazing comprehension and reasoning capabilities, semantic enrichment enhances reasoning performance, node ordering impacts task success, and the ability to process longer texts does not necessarily improve graph comprehension or reasoning.GraCoRe is open-sourced at https://github.com/ZIKEYUAN/GraCoRe
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）；IBM Research AI（IBM研究院人工智能部门）；Qassim University（卡西姆大学）
PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；WeChat AI, Tencent Inc., China（微信AI，腾讯公司，中国）
CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models
Tsinghua University（清华大学）；Jilin University（吉林大学）；Northwest Minzu University（西北民族大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
HIVMedQA: Benchmarking large language models for HIV medical decision support
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland（1 生物系统科学与工程系，苏黎世联邦理工学院，巴塞尔，瑞士）；School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland（2 生命科学学院，洛桑联邦理工学院，洛桑，瑞士）；Swiss Institute of Bioinformatics, Lausanne, Switzerland（3 瑞士生物信息学研究所，洛桑，瑞士）；Biomedical Data Science Center, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland（4 生物医学数据科学中心，洛桑大学医院和洛桑大学，洛桑，瑞士）；Institute of Medical Virology, University of Zurich, Zurich, Switzerland（5 医学病毒学研究所，苏黎世大学，苏黎世，瑞士）；Department of Infectious Diseases and Hospital Epidemiology, University Hospital Zurich, Zurich, Switzerland（6 感染病与医院流行病学系，苏黎世大学医院，苏黎世，瑞士）；ETH AI Center, ETH Zurich, Zurich, Switzerland（7 ETH人工智能中心，苏黎世联邦理工学院，苏黎世，瑞士）；Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland（8 量化生物医学系，苏黎世大学，苏黎世，瑞士）
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能学院）；Rochester Institute of Technology（罗切斯特理工学院）；Meta AI；Amazon Gen AI（亚马逊生成式人工智能部门）
Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning
Department of Medical Engineering（医学工程系）；Electrical Engineering Department（电气工程系）；Sharif University of Technology（谢里夫技术大学）；Department of Electrical Engineering（电气工程系）；University of Tehran（德黑兰大学）
ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following
Graduate School of Informatics, Kyoto University, Japan（京都大学信息科学研究生院）；College of Computer Science and Artificial Intelligence, Fudan University, China（复旦大学计算机科学与人工智能学院）；Graduate School of Engineering, Kyoto University, Japan（京都大学工学研究科）
MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；National Taiwan University（国立台湾大学）；Massachusetts Institute of Technology（麻省理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Meta AI
MatTools: Benchmarking Large Language Models for Materials Science Tools
Center for Structural Materials, Department of Mechanical Engineering, The University of Hong Kong（香港大学机械工程系结构材料中心）；Materials Innovation Institute for Life Sciences（生命科学创新研究所）
MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
University of California, Los Angeles Computer Science Department（加州大学洛杉矶分校计算机科学系）；University of Pennsylvania Department of Materials Science and Engineering（宾夕法尼亚大学材料科学与工程系）；Virginia Tech Department of Computer Science（弗吉尼亚理工大学计算机科学系）
TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Microsoft Research（微软研究院）；Ohio State University（俄亥俄州立大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Northeastern University（东北大学）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Chinese University of Hong Kong（香港中文大学）；Illinois Institute of Technology（伊利诺伊理工学院）；Georgia Institute of Technology（佐治亚理工学院）
提出 TemporalBench 基准，通过约 10K 高质量人工标注视频问答对评估多模态视频模型的细粒度时序理解能力，发现 GPT-4o 准确率仅 38.5%，并提出 MBA 指标纠正多选问答偏差。
OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
The Chinese University of Hong Kong（香港中文大学）；Alibaba Token Hub, Alibaba Group（阿里巴巴集团通义实验室）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Zhejiang University（浙江大学）
