Characterizing the patterns of errors that a system makes helps researchers focus future development on increasing its accuracy and robustness. We propose a novel form of "meta learning" that automatically learns interpretable rules that characterize the types of errors that a system makes, and demonstrate these rules' ability to help understand and improve two NLP systems. Our approach works by collecting error cases on validation data, extracting meta-features describing these samples, and finally learning rules that characterize errors using these features. We apply our approach to VilBERT, for Visual Question Answering, and RoBERTa, for Common Sense Question Answering. Our system learns interpretable rules that provide insights into systemic errors these systems make on the given tasks. Using these insights, we are also able to "close the loop" and modestly improve performance of these systems.
Towards Automated Error Discovery: A Study in Conversational AI
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt（德累斯顿技术大学）
SmartHomeSecure: Automated Detection and Repair of Smart Home Configuration Errors Using Large Language Models
Myers-Lawson School of Construction, Virginia Polytechnic Institute and State University（Myers-Lawson工程学院，弗吉尼亚理工学院和州立大学）；School of Technology, Eastern Illinois University（技术学院，东伊利诺伊大学）
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
We present a new approach to evaluate computational models for the task of text understanding by the means of out-of-context error detection. Through the novel design of our automated modification process, existing large-scale data sources can be adopted for a vast number of text understanding tasks. The data is thereby altered on a semantic level, allowing models to be tested against a challenging set of modified text passages that require to comprise a broader narrative discourse. Our newly introduced task targets actual real-world problems of transcription and translation systems by inserti…
ALTA Institute（ALTA研究所）；The Computer Laboratory（计算机实验室）
Grammatical error detection and automated essay scoring are two tasks in the area of automated assessment. Traditionally these tasks have been treated independently with different machine learning models and features used for each task. In this paper, we develop a multi-task neural network model that jointly optimises for both tasks, and in particular we show that neural automated essay scoring can be significantly improved. We show that while the essay score provides little evidence to inform grammatical error detection, the essay score is highly influenced by error detection.
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Wenge Technology（文英科技）
Masked language models have revolutionized natural language processing systems in the past few years. A recently introduced generalization of masked language models called warped language models are trained to be more robust to the types of errors that appear in automatic or manual transcriptions of spoken language by exposing the language model to the same types of errors during training. In this work we propose a novel approach that takes advantage of the robustness of warped language models to transcription noise for correcting transcriptions of spoken language. We show that our proposed ap…
RIKEN（理化学研究所）；Allen Institute for AI（艾伦人工智能研究所）；Earth Species Project（地球物种项目）；Octanove Labs（Octanove实验室）；Tohoku University（东北大学）
Natural language processing technology has rapidly improved automated grammatical error correction tasks, and the community begins to explore document-level revision as one of the next challenges. To go beyond sentence-level automated grammatical error correction to NLP-based document-level revision assistant, there are two major obstacles: (1) there are few public corpora with document-level revisions being annotated by professional editors, and (2) it is not feasible to elicit all possible references and evaluate the quality of revision with such references because there are infinite possibi…
Grammatical Error Correction (GEC) systems perform a sequence-to-sequence task, where an input word sequence containing grammatical errors, is corrected for these errors by the GEC system to output a grammatically correct word sequence. With the advent of deep learning methods, automated GEC systems have become increasingly popular. For example, GEC systems are often used on speech transcriptions of English learners as a form of assessment and feedback - these powerful GEC systems can be used to automatically measure an aspect of a candidate's fluency. The count of \textit{edits} from a candid…
Classification errors distort findings in automated speech processing: examples and solutions from child-development research
University of Missouri（密苏里大学）；University of Wuppertal（伍珀塔尔大学）；Laboratoire de Sciences Cognitives et Psycholinguistique（认知科学与心理语言学实验室）；EHESS（法国高等社会科学研究院）；CNRS（法国国家科学研究中心）；PSL University（巴黎文理研究大学）；Australian National University（澳大利亚国立大学）
Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems
Tohoku University, Sendai, Japan（东京东京大学）；RIKEN, Wako, Japan（日本理化学研究所）；ALTA Institute, Computer Laboratory, University of Cambridge, U.K（剑桥大学计算机实验室ALTA研究所）；INSA Lyon, Villeurbanne, France（里昂国立应用科学学院）；CNRS, LIS, Aix-Marseille University, Marseille, France（国家科学研究中心（CNRS））；MBZUAI, Abu Dhabi, United Arab Emirates（阿布扎赫德人工智能研究所）
Institute of Health Informatics, University of Minnesota（明尼苏达大学健康信息学研究所）；Biomedical Informatics and Medical Education, University of Washington（华盛顿大学生物医学信息学与医学教育系）；Department of Pharmaceutical Care & Health Systems, University of Minnesota（明尼苏达大学药物护理与卫生系统系）
Grammar checking is the task of detection and correction of grammatical errors in the text. English is the dominating language in the field of science and technology. Therefore, the non-native English speakers must be able to use correct English grammar while reading, writing or speaking. This generates the need of automatic grammar checking tools. So far many approaches have been proposed and implemented. But less efforts have been made in surveying the literature in the past decade. The objective of this systematic review is to examine the existing literature, highlighting the current issues…
