Recent NLP tasks have benefited a lot from pre-trained language models (LM) since they are able to encode knowledge of various aspects. However, current LM evaluations focus on downstream performance, hence lack to comprehensively inspect in which aspect and to what extent have they encoded knowledge. This paper addresses both queries by proposing four tasks on syntactic, semantic, commonsense, and factual knowledge, aggregating to a total of $39,308$ questions covering both linguistic and world knowledge in Chinese. Throughout experiments, our probes and knowledge data prove to be a reliable benchmark for evaluating pre-trained Chinese LMs. Our work is publicly available at https://github.com/ZhiruoWang/ChnEval.
East China Normal University（华东师范大学）；Alibaba Group（阿里巴巴集团）；Chongqing University（重庆大学）
Recently, knowledge-enhanced pre-trained language models (KEPLMs) improve context-aware representations via learning from structured relations in knowledge graphs, and/or linguistic knowledge from syntactic or dependency analysis. Unlike English, there is a lack of high-performing open-source Chinese KEPLMs in the natural language processing (NLP) community to support various language understanding applications. In this paper, we revisit and advance the development of Chinese natural language understanding with a series of novel Chinese KEPLMs released in various parameter sizes, namely CKBERT…
A Large-Scale Chinese Knowledge Graph-Text Alignment Dataset for Benchmarking Knowledge-Grounded LLMs
Nanjing University of Science and Techonolgy（南京理工大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Science and Technology Beijing（北京科技大学）；Hefei University of Technology（合肥工业大学）；Beijing University of Chemical Technology（北京化工大学）；Renmin University of China（中国人民大学）；Beihang University（北航）；Beijing University of Posts and Telecommunications（北京邮电大学）；Griffith University, Australia（澳大利亚格里菲斯大学）
State Key Laboratory of Media Convergence Production Technology and Systems（媒体融合生产技术与系统国家重点实验室）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）
Chinese short text matching is a fundamental task in natural language processing. Existing approaches usually take Chinese characters or words as input tokens. They have two limitations: 1) Some Chinese words are polysemous, and semantic information is not fully utilized. 2) Some models suffer potential issues caused by word segmentation. Here we introduce HowNet as an external knowledge base and propose a Linguistic knowledge Enhanced graph Transformer (LET) to deal with word ambiguity. Additionally, we adopt the word lattice graph as input to maintain multi-granularity information. Our model…
Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking
Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning（湖北省人工智能与智慧学习重点实验室）；National Language Resources Monitoring and Research Center for Network Media（国家网络媒体语言资源监测与研究中心）；School of Computer, Central China Normal University（华中师范大学计算机学院）；School of Computer Science and Information Engineering, Hubei University（湖北大学计算机与信息工程学院）
FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models
Shanghai University of Finance and Economics（上海财经大学）；University of California, Irvine（加州大学欧文分校）；Fudan University（复旦大学）；Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；Tencent YouTu Lab（腾讯优图实验室）；HSBC Lab（汇丰实验室）
Acquiring Common Chinese Emotional Events Using Large Language Model
School of Artificial Intelligence and Software Engineering, Nanyang Normal University（南阳师范学院人工智能与软件工程学院）；Key Laboratory of Intelligent Information Processing, Institute of Computer Technology, Chinese Academy of Sciences（中国科学院计算技术研究所智能信息处理重点实验室）
East China Normal University（东华师范大学）；Zhejiang University（浙江大学）；Imperial College London（伦敦帝国学院）
With the rapid development of large language models (LLMs), various LLM-based works have been widely applied in educational fields. However, most existing LLMs and their benchmarks focus primarily on the knowledge dimension, largely neglecting the evaluation of cultivation capabilities that are essential for real-world educational scenarios. Additionally, current benchmarks are often limited to a single subject or question type, lacking sufficient diversity. This issue is particularly prominent within the Chinese context. To address this gap, we introduce OmniEduBench, a comprehensive Chinese …
University of California, San Diego（加州大学圣迭戈分校）；Microsoft Research Asia（微软亚洲研究院）；Beihang University（北京航空航天大学）
Cant is important for understanding advertising, comedies and dog-whistle politics. However, computational research on cant is hindered by a lack of available datasets. In this paper, we propose a large and diverse Chinese dataset for creating and understanding cant from a computational linguistics perspective. We formulate a task for cant understanding and provide both quantitative and qualitative analysis for tested word embedding similarity and pretrained language models. Experiments suggest that such a task requires deep language understanding, common sense, and world knowledge and thus ca…
East China Normal University（华东师范大学）；Alibaba Group（阿里巴巴集团）；DXY（丁香园）
Machine Reading Comprehension (MRC) aims to extract answers to questions given a passage. It has been widely studied recently, especially in open domains. However, few efforts have been made on closed-domain MRC, mainly due to the lack of large-scale training data. In this paper, we introduce a multi-target MRC task for the medical domain, whose goal is to predict answers to medical questions and the corresponding support sentences from medical information sources simultaneously, in order to ensure the high reliability of medical knowledge serving. A high-quality dataset is manually constructe…
LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artifical Intelligence, Nanjing University（南京大学人工智能学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent Cloud Xiaowei（腾讯云小微）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Peng Cheng Laboratory（鹏城实验室）
Chinese Spell Checking (CSC) aims to detect and correct Chinese spelling errors, which are mainly caused by the phonological or visual similarity. Recently, pre-trained language models (PLMs) promote the progress of CSC task. However, there exists a gap between the learned knowledge of PLMs and the goal of CSC task. PLMs focus on the semantics in text and tend to correct the erroneous characters to semantically proper or commonly used ones, but these aren't the ground-truth corrections. To address this issue, we propose an Error-driven COntrastive Probability Optimization (ECOPO) framework for…
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine（上海中医药大学针灸推拿学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；China Academy of Chinese Medical Sciences（中国中医科学院）
Recent work has shown that Pre-trained Language Models (PLMs) store the relational knowledge learned from data and utilize it for performing downstream tasks. However, commonsense knowledge across different regions may vary. For instance, the color of bridal dress is white in American weddings whereas it is red in Chinese weddings. In this paper, we introduce a benchmark dataset, Geo-Diverse Commonsense Multilingual Language Models Analysis (GeoMLAMA), for probing the diversity of the relational knowledge in multilingual PLMs. GeoMLAMA contains 3,125 prompts in English, Chinese, Hindi, Persian…
TCM-DiffRAG: Personalized Syndrome Differentiation Reasoning Method for Traditional Chinese Medicine based on Knowledge Graph and Chain of Thought
Macao Polytechnic University（澳门理工大学）；Guangdong Institute of Intelligence Science and Technology（广东智能科学技术研究院）；Zhejiang Chinese Medical University（浙江中医药大学）；Hangzhou Ganzhicao Technology Co., Ltd（杭州甘之草科技有限公司）；BoardWare Information System Limited（BoardWare信息系统有限公司）
CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Jilin University（吉林大学）；University of Science and Technology of China（中国科学技术大学）；Shenzhen University of Advanced Technology（深圳理工大学）；University of Macau（澳门大学）；Huazhong University of Science and Technology（华中科技大学）
Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570GB training data, drew a lot of attention due to the capacity of few-shot (even zero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best of our knowl…
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications
Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Tencent Youtu Lab（腾讯优图实验室）；Cognition and Intelligent Technology Laboratory（认知与智能技术实验室）；Engineering Research Center of AI and Robotics, Ministry of Education（教育部人工智能与机器人工程研究中心）；AI and Unmanned Systems Engineering Research Center of Jilin Province（吉林省人工智能与无人系统工程研究中心）
Quantifying Self-diagnostic Atomic Knowledge in Chinese Medical Foundation Model: A Computational Analysis
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）
Hefei University of Technology（合肥工业大学）；Mininglamp Academy of Sciences（明略科学院）
Lexical simplification has attracted much attention in many languages, which is the process of replacing complex words in a given sentence with simpler alternatives of equivalent meaning. Although the richness of vocabulary in Chinese makes the text very difficult to read for children and non-native speakers, there is no research work for Chinese lexical simplification (CLS) task. To circumvent difficulties in acquiring annotations, we manually create the first benchmark dataset for CLS, which can be used for evaluating the lexical simplification systems automatically. In order to acquire more…
E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Science and Technology of China（中国科学技术大学）；Southern University of Science and Technology（南方科技大学）；UNION INFORMATION（联信信息）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
We present a simple yet effective method for generating high quality classical Chinese poetry with Generative Pre-trained Language Model (GPT). The method adopts a simple GPT model, without using any human crafted rules or features, or designing any additional neural components. While the proposed model learns to generate various forms of classical Chinese poems, including Jueju, Lüshi, various Cipai and Couples, the generated poems are of very high quality. We also propose and implement a method to fine-tune the model to generate acrostic poetry. To the best of our knowledge, this is the firs…
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University（华东师范大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Hasso Plattner Institute（哈索·普拉特纳研究所）；University of Potsdam（波茨坦大学）；Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作媒体网络创新中心）
MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
School of Software Engineering, Sun Yat-sen University（中山大学软件学院）；Institute of TCM Diagnostics, Hunan University of Chinese Medicine（湖南中医药大学中医诊断研究所）；School of Artificial Intelligence, Guilin University of Electronic Technology（桂林电子科技大学人工智能学院）；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）
Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry
NLP 2 CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学）；Department of Portuguese, Faculty of Arts and Humanities, University of Macau（文学与人文学院，澳门大学）
Chinese Essay Rhetoric Recognition Using LoRA, In-context Learning and Model Ensemble
The Open University of China（中国开放大学）；Engineering Research Center of Integration and Application of Digital Learning Technology, Ministry of Education（教育部数字学习技术整合与应用工程研究中心）
Harbin Institute of Technology（哈尔滨工业大学）；iFLYTEK Research（讯飞研究院）
Existing Chinese text error detection mainly focuses on spelling and simple grammatical errors. These errors have been studied extensively and are relatively simple for humans. On the contrary, Chinese semantic errors are understudied and more complex that humans cannot easily recognize. The task of this paper is Chinese Semantic Error Recognition (CSER), a binary classification task to determine whether a sentence contains semantic errors. The current research has no effective method to solve this task. In this paper, we inherit the model structure of BERT and design several syntax-related pr…
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community（多模态艺术投影研究共同体）；Hong Kong University of Science and Technology（香港科技大学）；University of Waterloo（滑铁卢大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peking University（北京大学）；The Hong Kong Polytechnic University（香港理工大学）；Waseda University（早稻田大学）；University of Manchester（曼彻斯特大学）
Existing commonsense knowledge bases often organize tuples in an isolated manner, which is deficient for commonsense conversational models to plan the next steps. To fill the gap, we curate a large-scale multi-turn human-written conversation corpus, and create the first Chinese commonsense conversation knowledge graph which incorporates both social commonsense knowledge and dialog flow information. To show the potential of our graph, we develop a graph-conversation matching approach, and benchmark two graph-grounded conversational tasks.
Ping An Life Insurance of China（中国平安人寿保险股份有限公司）；Fudan University（复旦大学）
For the task of open domain Knowledge Based Question Answering in CCKS2019, we propose a method combining information retrieval and semantic parsing. This multi-module system extracts the topic entity and the most related relation predicate from a question and transforms it into a Sparql query statement. Our method obtained the F1 score of 70.45% on the test data.
Recently, many studies incorporate external knowledge into character-level feature based models to improve the performance of Chinese relation extraction. However, these methods tend to ignore the internal information of the Chinese character and cannot filter out the noisy information of external knowledge. To address these issues, we propose a mixture-of-view-experts framework (MoVE) to dynamically learn multi-view features for Chinese relation extraction. With both the internal and external knowledge of Chinese characters, our framework can better capture the semantic information of Chinese…
