To thoroughly assess the mathematical reasoning abilities of Large Language Models (LLMs), we need to carefully curate evaluation datasets covering diverse mathematical concepts and mathematical problems at different difficulty levels. In pursuit of this objective, we propose FineMath in this paper, a fine-grained mathematical evaluation benchmark dataset for assessing Chinese LLMs. FineMath is created to cover the major key mathematical concepts taught in elementary school math, which are further divided into 17 categories of math word problems, enabling in-depth analysis of mathematical reasoning abilities of LLMs. All the 17 categories of math word problems are manually annotated with their difficulty levels according to the number of reasoning steps required to solve these problems. We conduct extensive experiments on a wide range of LLMs on FineMath and find that there is still considerable room for improvements in terms of mathematical reasoning capability of Chinese LLMs. We also carry out an in-depth analysis on the evaluation process and methods that have been overlooked previously. These two factors significantly influence the model results and our understanding of their mathematical reasoning capabilities. The dataset will be publicly available soon.
ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
ChiMDQA: Towards Comprehensive Chinese Document QA with Fine-grained Evaluation
Beijing Jiaotong University(北京交通大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing University of Technology(北京理工大学);Foxit Software Co. Ltd(Foxit软件有限公司)
CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment
Tsinghua University(清华大学);Beijing Normal University(北京师范大学);South China University of Technology(华南理工大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Shenzhen ShenNong Information Technology Co., Ltd.(深圳神农信息技术有限公司)
SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis
School of Software Engineering, Beijing University of Technology(北京工业大学软件学院);School of Nursing, Wuhan University(武汉大学护理学院);Sorbonne Université(索邦大学);Institut du Cerveau - Paris Brain Institute - ICM(大脑研究所-巴黎脑研究所-ICM);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);Inserm(法国国家健康与医学研究院);AP-HP(巴黎公共医疗集团);Hôpital de la Pitié Salpêtrière(皮蒂-萨尔皮特里医院)
Towards Comprehensive Detection of Chinese Harmful Memes
School of Computer Science and Technology(计算机科学与技术学院);School of Foreign Languages(外国语学院);Key Laboratory of Social Computing and Cognitive Intelligence(社会计算与认知智能重点实验室);Dalian University of Technology(大连理工大学)
Classic4Children: Adapting Chinese Literary Classics for Children with Large Language Model
Key Laboratory of Big Data and Intelligent Robot (South China University of Technology) Ministry of Education(大数据与智能机器人重点实验室(华南理工大学)教育部);School of Software Engineering, South China University of Technology(软件学院(华南理工大学));The Hong Kong Polytechnic University(香港理工大学)
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University(浙江大学ZJU-UIUC学院);University of Washington(华盛顿大学);Zhejiang University School of Medicine Sir Run Run Shaw Hospital(浙江大学医学院附属邵逸夫医院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
CNSocialDepress: A Chinese Social Media Dataset for Depression Risk Detection and Structured Analysis
Ertim Inalco;Milkuya Studio;Sorbonne Université(索邦大学);IRD Lab(IRD实验室);Faculty of Psychology, Peking University(北京大学心理学系);Faculty of Psychology and Cognitive Science, Beijing Normal University(北京师范大学心理学与认知科学系);Beijing Institute of Technology(北京理工大学)
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
School of Informatics, Xiamen University, China(厦门大学信息学院);Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China(福建省-台湾非物质文化遗产数字化保护与智能处理重点实验室(XMU),文化和旅游部,中国);National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China(教育与教学媒体语言资源监测与研究中心,厦门大学,中国)
ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Model
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Wuhan AI Research(武汉人工智能研究院)
CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark
National Key Laboratory for Multimedia Information Processing(多媒体信息处理国家重点实验室);Peking University School of Computer Science(北京大学计算机科学学院);Peking University School of Software and Microelectronics(北京大学软件与微电子学院)
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
University of International Relations(国际关系学院);Kedge Business School(凯致商学院);Peking University(北京大学);Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);Capital Normal University(首都师范大学)
SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);University of Liverpool(利物浦大学);City University of Hong Kong(香港城市大学);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);China Mobile Research Institute(中国移动研究院);China Mobile (Suzhou) Software Technology Co., Ltd(中国移动苏州软件技术有限公司)
EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors
School of Software(软件学院);Dalian University of Technology(大连理工大学);School of Foreign Languages and School of Software(外语学院和软件学院);Faculty of Business Administration(商学院);University of Macau(澳门大学);School of Computing Technologies(计算技术学院);RMIT University(皇家墨尔本理工大学)