For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors (typos). Thus, it is crucial to test model performance on data with realistic input noises to ensure robustness and fairness. However, little study has been done to construct such benchmarks for Chinese, where various language-specific input noises happen in the real world. In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We designed our annotation pipeline to maximize diversity, for example by instructing the annotators to use diverse input method editors (IMEs) for keyboard noises and recruiting speakers from diverse dialectical groups for speech noises. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN even with robustness methods like data augmentation. As the first large-scale attempt in creating a benchmark with noises geared towards user-generated inputs, we believe that READIN serves as an important complement to existing Chinese NLP benchmarks. The source code and dataset can be obtained from https://github.com/thunlp/READIN.
Consensus Attention-based Neural Networks for Chinese Reading Comprehension
iFLYTEK Research(科大讯飞研究);Research Center for Social Computing and Information Retrieval,Harbin Institute of Technology(哈尔滨工业大学社会计算与信息检索研究中心);Harbin Institute of Technology(哈尔滨工业大学)
ChID: A Large-scale Chinese IDiom Dataset for Cloze Test
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Nanyang Technological University(南洋理工大学)
Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents
Stanford University(斯坦福大学);Université Paris-Saclay(巴黎-萨克雷大学);CEA(法国原子能和替代能源委员会);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);International Institute of Information Technology, Hyderabad(海得拉巴国际信息技术学院);Hanyang University(汉阳大学);Tianjin University(天津大学)
Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);IDEA(深圳国际数字经济研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
SinoGlyphBench: A Diagnostic Benchmark for Chinese Glyph-Level Obfuscation in Language-Model Moderation
East China Normal University(华东师范大学);New York University Shanghai(上海纽约大学);University of Liverpool(利物浦大学);Singapore University of Technology and Design(新加坡科技设计大学);Eastern Institute of Technology (Ningbo)(宁波东方理工大学)
A realistic and robust model for Chinese word segmentation
Institute of Linguistics, Academia Sinica(中央研究院语言学研究所);Hong Kong Polytechnic University(香港理工大学);Academia Sinica(中央研究院);National Taiwan Normal University(台湾师范大学)
MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Institute of Infectious Disease and Biosecurity, Fudan University(复旦大学上海传染病与生物安全研究院);Shanghai Health Development Research Center (Shanghai Medical Information Center)(上海市卫生和健康发展研究中心(上海市医学科学技术情报研究所));Imperial College London(伦敦帝国理工学院)
LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis
Tianqiao and Chrissy Chen Institute(天桥和克里斯西·陈研究所);EverMind AI Inc.(EverMind AI公司);Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine(上海精神卫生中心,上海交通大学医学院)