We present the Chinese Elementary School Math Word Problems (CMATH) dataset, comprising 1.7k elementary school-level math word problems with detailed annotations, source from actual Chinese workbooks and exams. This dataset aims to provide a benchmark tool for assessing the following question: to what grade level of elementary school math do the abilities of popular large language models (LLMs) correspond? We evaluate a variety of popular LLMs, including both commercial and open-source options, and discover that only GPT-4 achieves success (accuracy $\geq$ 60\%) across all six elementary school grades, while other models falter at different grade levels. Furthermore, we assess the robustness of several top-performing LLMs by augmenting the original problems in the CMATH dataset with distracting information. Our findings reveal that GPT-4 is able to maintains robustness, while other model fail. We anticipate that our study will expose limitations in LLMs' arithmetic and reasoning capabilities, and promote their ongoing development and advancement.
E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Science and Technology of China(中国科学技术大学);Southern University of Science and Technology(南方科技大学);UNION INFORMATION(联信信息);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);University of Edinburgh(爱丁堡大学);The Hong Kong University of Science and Technology(香港科技大学)
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension
IIIS, Tsinghua University(清华大学交叉信息研究院);New York University(纽约大学);Haihua Institute for Frontier Information Technology(海华前沿信息技术研究院);Shenzhen University(深圳大学);Peking University(北京大学);Shanghai Qi Zhi Institute(上海期智研究院)
我们提出了Native Chinese Reader(NCR),一个新的机器阅读理解(MRC)数据集,特别是现代和古典汉语的长文章。NCR是从中国高中语文课程的考试试题中收集的,旨在评估中国本土青年的语言能力。现有的中文MRC数据集要么是特定领域的,要么只关注现代汉语中几百个字符的短上下文。相比之下,NCR包含8390个文档,平均长度为1024个字符,涵盖了广泛的中文写作风格,包括现代文章,古典文学和古典诗歌。这些文件上总共有20477道题,也需要很强的推理能力和常识才能算出正确答案。我们使用流行的中国预训练模型实现了多个基线模型,并使用我们的数据集发起了一场在线竞赛,以检查当前方法的局限性。最好的模型达到了59%的测试准确率,而人工评估的平均准确率为79%,这表明当前MRC模型与母语为汉语的人之间存在显着的性能差距。我们在https://sites.google.com/view/native-chinese-reader/上发布数据集。
Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
Lehigh University(里海大学);Squirrel Ai Learning(松鼠AI学习);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Michigan State University(密歇根州立大学)
CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);East China Normal University(华东师范大学);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室)
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset
ShanghaiTech University(上海科技大学);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心);University of Science and Technology of China(中国科学技术大学);Shanghai Innovation Center for Processor Technologies(上海处理器技术创新中心);Shanghai Jiao Tong University(上海交通大学);National Engineering Laboratory for Brain-inspired Intelligence Technology and Application(国家类脑智能技术及应用工程实验室);Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education(脑与认知科学教育部重点实验室)