CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark
Tsinghua University(清华大学);Peking University(北京大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);JD AI Research(京东人工智能研究院);Beijing Language and Culture University(北京语言大学);Renmin University of China(中国人民大学)
We add two new datasets, including grammatical error correction dataset YACLC from Beijing Language and Culture University, and reading comprehension dataset GCRC from Shanxi University, and also improve the description consistency of all datasets
Realizing general-purpose language intelligence has been a longstanding goal for natural language processing, where standard evaluation benchmarks play a fundamental and guiding role. We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic. To this end, we propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features: (1) Hierarchical benchmark framework, where datasets are principally selected and organized with a language capability-task-dataset hierarchy. (2) Multi-level scoring strategy, where different levels of model performance are provided based on the hierarchical framework. To facilitate CUGE, we provide a public leaderboard that can be customized to support flexible model judging criteria. Evaluation results on representative pre-trained language models indicate ample room for improvement towards general-purpose language intelligence. CUGE is publicly available at cuge.baai.ac.cn.
YACLC: A Chinese Learner Corpus with Multidimensional Annotation
Beijing Language and Culture University(北京语言大学);Beijing Normal University(北京师范大学);Yunnan Normal University(云南师范大学);Northeastern University(东北大学);Shanghai University of Finance and Economics(上海财经大学);Tsinghua University(清华大学)
学习者语料库收集第二语言学习者,即第二语言或外语学习者的语言数据。该资源对第二语言习得研究、外语教学和自动语法纠错具有重要意义。然而,针对对外汉语学习者的学习者语料库研究却很少。因此,我们建议建立一个大规模、多维的汉语学习者语料库。为了构建语料库,我们首先获得了大量的由汉语学习者生成的富文本。然后,我们设计了一个注释方案,包括句子的可接受性评分以及语法错误和流畅性为基础的更正。我们建立了一个众包平台来有效地执行注释(yaclc.wenmind.net)。我们将该语料库命名为YACLC(Yet Another Chinese Learner Corpus),并将其作为CUGE基准测试的一部分(http://www.example.com)发布。cuge.baai.ac.cn通过对语料库中的原文和注释的分析,我们发现YACLC具有相当大的规模和非常高的注释质量。希望该语料库能进一步促进汉语国际教育和汉语语法错误自动纠正的研究。
CERD: A Comprehensive Chinese Rhetoric Dataset for Rhetorical Understanding and Generation in Essays
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Shanghai Institute of AI for Education, East China Normal University(华东师范大学上海人工智能教育研究院);Department of Chinese Language and Literature, East China Normal University(华东师范大学中文系);Microsoft Research Asia(微软亚洲研究院)
LongHisDoc: A Comprehensive Benchmark for Chinese Long Historical Document Understanding
School of Electronic and Information Engineering, South China University of Technology(华南理工大学电子与信息工程学院);College of Literature and Journalism, Sichuan University(四川大学文学与新闻学院)
The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese
Princeton University(普林斯顿大学);Shanghai Jiao Tong University(上海交通大学);Chinese Academy of Sciences(中国科学院);Columbia University(哥伦比亚大学);Beijing Normal University(北京师范大学);Tsinghua University(清华大学);University of California San Diego(加利福尼亚大学圣地亚哥分校);The Hong Kong Polytechnic University(香港理工大学)
Distinguishing Right from Wrong in Debates: Attribution Analysis of Chinese Harmful Memes
Key Laboratory of Social Computing and Cognitive Intelligence, Dalian University of Technology(社会计算与认知智能重点实验室,大连理工大学);Singapore University of Technology and Design(新加坡科技设计大学);Data Intelligence Lab, Department of Computer Science, City University of Hong Kong(数据智能实验室,香港城市大学计算机科学系)
CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark
National Key Laboratory for Multimedia Information Processing(多媒体信息处理国家重点实验室);Peking University School of Computer Science(北京大学计算机科学学院);Peking University School of Software and Microelectronics(北京大学软件与微电子学院)
A Chinese Multi-label Affective Computing Dataset Based on Social Media Network Users
Beijing Institute of Technology(北京理工大学);Institute of Scientific and Technical Research on Archives(档案科学技术研究所);China Electronics Engineering Design Institute Co., Ltd.(中国电子工程设计院有限公司)
预训练语言模型在自然语言理解(NLU)任务中取得了巨大的成功,因为它能够通过对大规模语料库的预训练来捕获文本中的深层语境信息。在这份技术报告中,我们介绍了我们在中文语料库上预训练语言模型NEZHA(NEural contextualiZed representation for CHinese语言理解)的实践,并为中文NLU任务进行了微调。NEZHA的当前版本基于BERT,并进行了一系列改进,其中包括作为有效位置编码方案的功能相对位置编码,全字掩码策略,混合精度训练和LAMB优化器。实验结果表明,NEZHA在命名实体识别(人民日报NER)、句子匹配(LCQMC)、中文情感分类(ChnSenti)和自然语言推理(XNLI)等典型中文任务上均达到了最佳性能。
Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training
Peking University(北京大学);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室)
Evaluating Chinese Ambiguity Understanding in Large Language Models
Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院);School of Software Engineering, South China University of Technology(华南理工大学软件学院)
为了提高中文金融自然语言处理的效率,在T5语言模型的基础上提出了一种新的中文金融预训练语言模型BBT-FinT 5。为了支持这一努力,我们建立了BBT-FinCorpus,这是一个大规模的金融语料库,拥有来自四个不同来源的大约300 GB的原始文本。在一般领域的NLP中,像GLUE和SuperGLUE这样的综合基准测试通过在模型之间进行头对头比较,推动了语言模型预训练的重大进步。在此基础上,我们提出了一个中文金融语言理解与生成评估基准BBT-CFLEB,该基准包括六个数据集,涵盖理解和生成任务。我们的目标是促进中国金融领域NLP发展的研究。我们的模型、语料库和基准测试发布在https://github.com/ssymmetry/BBT-FinCUGE-Applications。我们的工作属于Big Bang Transformer(BBT),这是一个大规模的预训练语言模型项目。
PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain
East China Normal University(华东师范大学);Alibaba Group(阿里巴巴集团);Holoflow Digital Technology(全息数字技术(杭州)有限公司);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
University of International Relations(国际关系学院);Kedge Business School(凯致商学院);Peking University(北京大学);Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);Capital Normal University(首都师范大学)
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture
University of Copenhagen(哥本哈根大学);University of Waterloo(滑铁卢大学);Comcast AI Technologies(康卡斯特人工智能技术公司);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);University of Amsterdam(阿姆斯特丹大学)
Parsing Through Boundaries in Chinese Word Segmentation
The Chinese University of Hong Kong, Hong Kong(香港中文大学);University College London, United Kingdom(伦敦大学学院);The University of British Columbia, Canada(不列颠哥伦比亚大学)