Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset
Harvard University(哈佛大学);Peking University(北京大学);Second Affiliated Hospital of Zhejiang University School of Medicine(浙江大学医学院附属第二医院);Johns Hopkins University(约翰斯·霍普金斯大学);Yale University(耶鲁大学);Tianyi Traffic Technology(天翼交通科技);Ant Group(蚂蚁集团);Boston Children’s Hospital(波士顿儿童医院);Harvard Business School(哈佛商学院);Alibaba Group(阿里巴巴集团);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, Michael Lingzhi Li
备注:
Accepted by NeurIPS 2023 Datasets and Benchmarks Track
Recent advancements in large language models (LLMs) have transformed the field of question answering (QA). However, evaluating LLMs in the medical field is challenging due to the lack of standardized and comprehensive datasets. To address this gap, we introduce CMExam, sourced from the Chinese National Medical Licensing Examination. CMExam consists of 60K+ multiple-choice questions for standardized and objective evaluations, as well as solution explanations for model reasoning evaluation in an open-ended manner. For in-depth analyses of LLMs, we invited medical professionals to label five additional question-wise annotations, including disease groups, clinical departments, medical disciplines, areas of competency, and question difficulty levels. Alongside the dataset, we further conducted thorough experiments with representative LLMs and QA algorithms on CMExam. The results show that GPT-4 had the best accuracy of 61.6% and a weighted F1 score of 0.617. These results highlight a great disparity when compared to human accuracy, which stood at 71.6%. For explanation tasks, while LLMs could generate relevant reasoning and demonstrate improved performance after finetuning, they fall short of a desired standard, indicating ample room for improvement. To the best of our knowledge, CMExam is the first Chinese medical exam dataset to provide comprehensive medical annotations. The experiments and findings of LLM evaluation also provide valuable insights into the challenges and potential solutions in developing Chinese medical QA systems and LLM evaluation pipelines. The dataset and relevant code are available at https://github.com/williamliujl/CMExam.
Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model
Harvard T.H. Chan School of Public Health(哈佛陈曾熙公共卫生学院);Peking University(北京大学);University of Oxford(牛津大学);Alibaba Group(阿里巴巴集团);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Ant Group(蚂蚁集团);Tianyi Traffic Technology(天翼交通科技);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Jilin University(吉林大学);University of Science and Technology of China(中国科学技术大学);Shenzhen University of Advanced Technology(深圳理工大学);University of Macau(澳门大学);Huazhong University of Science and Technology(华中科技大学)
OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);School of New Media and Communication, Tianjin University(天津大学新媒体与传播学院);School of Computer and Artificial Intelligence, Zhengzhou University(郑州大学计算机与人工智能学院)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
The Chinese University of Hong Kong(香港中文大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));LightSpeed Studios, Tencent Inc.(腾讯公司光速工作室);Sun Yat-Sen University(中山大学)
MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan
Center for Integrated Japanese Studies, Tohoku University(东京东京大学综合日语研究机构);University of California, Berkeley(加州大学伯克利分校);Berkeley Artificial Intelligence Research (BAIR)(伯克利人工智能研究(BAIR))
CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);School of Computer and Artificial Intelligence, Zhengzhou University(郑州大学计算机与人工智能学院)
Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension
IIIS, Tsinghua University(清华大学交叉信息研究院);New York University(纽约大学);Haihua Institute for Frontier Information Technology(海华前沿信息技术研究院);Shenzhen University(深圳大学);Peking University(北京大学);Shanghai Qi Zhi Institute(上海期智研究院)
我们提出了Native Chinese Reader(NCR),一个新的机器阅读理解(MRC)数据集,特别是现代和古典汉语的长文章。NCR是从中国高中语文课程的考试试题中收集的,旨在评估中国本土青年的语言能力。现有的中文MRC数据集要么是特定领域的,要么只关注现代汉语中几百个字符的短上下文。相比之下,NCR包含8390个文档,平均长度为1024个字符,涵盖了广泛的中文写作风格,包括现代文章,古典文学和古典诗歌。这些文件上总共有20477道题,也需要很强的推理能力和常识才能算出正确答案。我们使用流行的中国预训练模型实现了多个基线模型,并使用我们的数据集发起了一场在线竞赛,以检查当前方法的局限性。最好的模型达到了59%的测试准确率,而人工评估的平均准确率为79%,这表明当前MRC模型与母语为汉语的人之间存在显着的性能差距。我们在https://sites.google.com/view/native-chinese-reader/上发布数据集。
RE$^2$: Improving Chinese Grammatical Error Correction via Retrieving Appropriate Examples with Explanation
Research Center for SCIR, Harbin Institute of Technology(SCIR研究中心,哈尔滨工业大学);State Key Laboratory of Cognitive Intelligence, iFLYTEK Research(认知智能国家重点实验室,iFLYTEK研究院);Artificial Intelligence and Human Language Lab, Beijing Foreign Studies University(人工智能与语言实验室,北京外国语大学)
Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training
Peking University(北京大学);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室)
Parsing Through Boundaries in Chinese Word Segmentation
The Chinese University of Hong Kong, Hong Kong(香港中文大学);University College London, United Kingdom(伦敦大学学院);The University of British Columbia, Canada(不列颠哥伦比亚大学)
CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)