arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.13610cs.CL
收藏

CUGE:一个中文语言理解和生成评估基准

CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark

Tsinghua University(清华大学);Peking University(北京大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);JD AI Research(京东人工智能研究院);Beijing Language and Culture University(北京语言大学);Renmin University of China(中国人民大学)
Yuan Yao, Qingxiu Dong, Jian Guan, Boxi Cao, Zhengyan Zhang, Chaojun Xiao, Xiaozhi Wang, Fanchao Qi, Junwei Bao, Jinran Nie, Zheni Zeng, Yuxian Gu, Kun Zhou, Xu… 展开作者

Yuan Yao, Qingxiu Dong, Jian Guan, Boxi Cao, Zhengyan Zhang, Chaojun Xiao, Xiaozhi Wang, Fanchao Qi, Junwei Bao, Jinran Nie, Zheni Zeng, Yuxian Gu, Kun Zhou, Xuancheng Huang, Wenhao Li, Shuhuai Ren, Jinliang Lu, Chengqiang Xu, Huadong Wang, Guoyang Zeng, Zile Zhou, Jiajun Zhang, Juanzi Li, Minlie Huang, Rui Yan, Xiaodong He, Xiaojun Wan, Xin Zhao, Xu Sun, Yang Liu, Zhiyuan Liu, Xianpei Han, Erhong Yang, Zhifang Sui, Maosong Sun

摘要:

实现通用语言智能是自然语言处理长期以来的目标,标准的评估基准在其中起着基础和指导作用。我们认为,对于通用语言智力评估,基准本身需要全面和系统。为此,我们提出了CUGE,一个中文语言理解和生成评估基准,它具有以下特点:(1)层次基准框架,其中数据集主要按照语言能力-任务-数据集的层次结构进行选择和组织。(2)多级评分策略,根据分层框架提供不同级别的模型性能。为了方便CUGE,我们提供了一个公共排行榜,可以自定义以支持灵活的模型判断标准。对具有代表性的预训练语言模型的评估结果表明,通用语言智能有很大的改进空间。CUGE可在cuge.baai.ac.cn公开获取。

英文摘要:

Realizing general-purpose language intelligence has been a longstanding goal for natural language processing, where standard evaluation benchmarks play a fundamental and guiding role. We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic. To this end, we propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features: (1) Hierarchical benchmark framework, where datasets are principally selected and organized with a language capability-task-dataset hierarchy. (2) Multi-level scoring strategy, where different levels of model performance are provided based on the hierarchical framework. To facilitate CUGE, we provide a public leaderboard that can be customized to support flexible model judging criteria. Evaluation results on representative pre-trained language models indicate ample room for improvement towards general-purpose language intelligence. CUGE is publicly available at cuge.baai.ac.cn.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑