Grammatical feedback is crucial for consolidating second language (L2) learning. Most research in computer-assisted language learning has focused on feedback through grammatical error correction (GEC) systems, rather than examining more holistic feedback that may be more useful for learners. This holistic feedback will be referred to as grammatical error feedback (GEF). In this paper, we present a novel implicit evaluation approach to GEF that eliminates the need for manual feedback annotations. Our method adopts a grammatical lineup approach where the task is to pair feedback and essay representations from a set of possible alternatives. This matching process can be performed by appropriately prompting a large language model (LLM). An important aspect of this process, explored here, is the form of the lineup, i.e., the selection of foils. This paper exploits this framework to examine the quality and need for GEC to generate feedback, as well as the system used to generate feedback, using essays from the Cambridge Learner Corpus.
Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing
National Institute of Standards（国家标准 institute）；Department of Physics, Colorado State University（物理系，科罗拉多州立大学）；Electrical Engineering Department, University of Colorado（电气工程系，科罗拉多大学）
Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems
Tohoku University, Sendai, Japan（东京东京大学）；RIKEN, Wako, Japan（日本理化学研究所）；ALTA Institute, Computer Laboratory, University of Cambridge, U.K（剑桥大学计算机实验室ALTA研究所）；INSA Lyon, Villeurbanne, France（里昂国立应用科学学院）；CNRS, LIS, Aix-Marseille University, Marseille, France（国家科学研究中心（CNRS））；MBZUAI, Abu Dhabi, United Arab Emirates（阿布扎赫德人工智能研究所）
My Words Imply Your Opinion: Reader Agent-based Propagation Enhancement for Personalized Implicit Emotion Analysis
School of Computer and Information Technology, Shanxi University, China（山西大学计算机与信息学院）；Institute of Automation, Chinese Academy of Science, China（中国科学院自动化研究所）；Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, Shanxi University, China（教育部计算智能与中文信息处理重点实验室）；Joint Laboratory of Tourism Big Data in Shanxi Province, China（山西省旅游大数据联合实验室）
UC Irvine（加州大学欧文分校）；Harvard University（哈佛大学）；Primate Research Institute, Kyoto University（京都大学灵长类研究所）；MIT（麻省理工学院）
Recurrent neural networks (RNNs) are the state of the art in sequence modeling for natural language. However, it remains poorly understood what grammatical characteristics of natural language they implicitly learn and represent as a consequence of optimizing the language modeling objective. Here we deploy the methods of controlled psycholinguistic experimentation to shed light on to what extent RNN behavior reflects incremental syntactic state and grammatical dependency representations known to characterize human linguistic behavior. We broadly test two publicly available long short-term memor…
George Washington University（乔治华盛顿大学）；University of Washington（华盛顿大学）
Does the grammatical gender of a language interfere when measuring the semantic gender information captured by its word embeddings? A number of anomalous gender bias measurements in the embeddings of gendered languages suggest this possibility. We demonstrate that word embeddings learn the association between a noun and its grammatical gender in grammatically gendered languages, which can skew social gender bias measurements. Consequently, word embedding post-processing methods are introduced to quantify, disentangle, and evaluate grammatical gender signals. The evaluation is performed on five…
Cambridge University Engineering Department（剑桥大学工程系）；Yandex（Yandex公司）
Ensemble approaches are commonly used techniques to improving a system by combining multiple model predictions. Additionally these schemes allow the uncertainty, as well as the source of the uncertainty, to be derived for the prediction. Unfortunately these benefits come at a computational and memory cost. To address this problem ensemble distillation (EnD) and more recently ensemble distribution distillation (EnDD) have been proposed that compress the ensemble into a single model, representing either the ensemble average prediction or prediction distribution respectively. This paper examines …
In this paper, we explore the capacity of a language model-based method for grammatical error detection in detail. We first show that 5 to 10% of training data are enough for a BERT-based error detection method to achieve performance equivalent to a non-language model-based method can achieve with the full training data; recall improves much faster with respect to training data size in the BERT-based method than in the non-language model method while precision behaves similarly. These suggest that (i) the BERT-based method should have a good knowledge of grammar required to recognize certain t…
RoLegalGEC: Legal Domain Grammatical Error Detection and Correction Dataset for Romanian
National University of Science and Technology POLITEHNICA Bucharest（波兰亚技术大学布加勒斯特分校）；Paris 1 Panthéon-Sorbonne University（巴黎1大学蓬莎博纳大学）；University of Bucharest（布加勒斯特大学）
A longstanding question in cognitive science concerns the learning mechanisms underlying compositionality in human cognition. Humans can infer the structured relationships (e.g., grammatical rules) implicit in their sensory observations (e.g., auditory speech), and use this knowledge to guide the composition of simpler meanings into complex wholes. Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is po…
News recommendation is different from movie or e-commercial recommendation as people usually do not grade the news. Therefore, user feedback for news is always implicit (click behavior, reading time, etc). Inevitably, there are noises in implicit feedback. On one hand, the user may exit immediately after clicking the news as he dislikes the news content, leaving the noise in his positive implicit feedback; on the other hand, the user may be recommended multiple interesting news at the same time and only click one of them, producing the noise in his negative implicit feedback. Opposite implicit…
Grammatical Error Correction (GEC) systems perform a sequence-to-sequence task, where an input word sequence containing grammatical errors, is corrected for these errors by the GEC system to output a grammatically correct word sequence. With the advent of deep learning methods, automated GEC systems have become increasingly popular. For example, GEC systems are often used on speech transcriptions of English learners as a form of assessment and feedback - these powerful GEC systems can be used to automatically measure an aspect of a candidate's fluency. The count of \textit{edits} from a candid…
Using Implicit Feedback to Improve Question Generation
Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；INESC-ID（葡萄牙信息与通信技术研究院）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
本文提出 GEN 系统，利用用户选择与修正两类隐式反馈迭代生成、评分并排序问题模式，实验表明其在 top 5、10、20 设置下较无学习版本提升至少 10%。
RDF2vec is a technique for creating vector space embeddings from an RDF knowledge graph, i.e., representing each entity in the graph as a vector. It first creates sequences of nodes by performing random walks on the graph. In a second step, those sequences are processed by the word2vec algorithm for creating the actual embeddings. In this paper, we explore the use of external edge weights for guiding the random walks. As edge weights, transition probabilities between pages in Wikipedia are used as a proxy for the human feedback for the importance of an edge. We show that in some scenarios, RDF…
Natural Language Understanding (NLU) is an established component within a conversational AI or digital assistant system, and it is responsible for producing semantic understanding of a user request. We propose a scalable and automatic approach for improving NLU in a large-scale conversational AI system by leveraging implicit user feedback, with an insight that user interaction data and dialog context have rich information embedded from which user satisfaction and intention can be inferred. In particular, we propose a general domain-agnostic framework for curating new supervision data for impro…
Improved Personalized Headline Generation via Denoising Fake Interests from Implicit Feedback
Henan Institute of Advanced Technology, Zhengzhou University（河南高级技术研究所，郑州大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Independent Researcher（独立研究者）
Training and refreshing a web-scale Question Answering (QA) system for a multi-lingual commercial search engine often requires a huge amount of training examples. One principled idea is to mine implicit relevance feedback from user behavior recorded in search engine logs. All previous works on mining implicit relevance feedback target at relevance of web documents rather than passages. Due to several unique characteristics of QA tasks, the existing user behavior models for web documents cannot be applied to infer passage relevance. In this paper, we make the first study to explore the correlat…
Nanjing University（南京大学）；Nanjing University of Science and Technology（南京理工大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Florida International University（佛罗里达国际大学）
Recommender systems (RSs) provide an effective way of alleviating the information overload problem by selecting personalized items for different users. Latent factors based collaborative filtering (CF) has become the popular approaches for RSs due to its accuracy and scalability. Recently, online social networks and user-generated content provide diverse sources for recommendation beyond ratings. Although {\em social matrix factorization} (Social MF) and {\em topic matrix factorization} (Topic MF) successfully exploit social relations and item reviews, respectively, both of them ignore some us…
Learning from Implicit User Feedback, Emotions and Demographic Information in Task-Oriented and Document-Grounded Dialogues
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；Department of Computer Science（计算机科学系）；Ubiquitous Knowledge Processing Lab (UKP Lab)（普适知识处理实验室）
Univ. Grenoble Alpes/CNRS（格勒诺布尔阿尔卑斯大学/法国国家科学研究中心）；Center for Energy Systems, Skolkovo Institute of Science（斯科尔科沃科技学院能源系统中心）；T-4 and CNLS, Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室T-4与CNLS）；Center for Energy Systems, Skolkovo Institute of Science and Technology（斯科尔科沃科技学院能源系统中心）
In this paper, we propose a novel ranking framework for collaborative filtering with the overall aim of learning user preferences over items by minimizing a pairwise ranking loss. We show the minimization problem involves dependent random variables and provide a theoretical analysis by proving the consistency of the empirical risk minimization in the worst case where all users choose a minimal number of positive and negative items. We further derive a Neural-Network model that jointly learns a new representation of users and items in an embedded space as well as the preference relation of user…
We present the first real-world application of methods for improving neural machine translation (NMT) with human reinforcement, based on explicit and implicit user feedback collected on the eBay e-commerce platform. Previous work has been confined to simulation experiments, whereas in this paper we work with real logged feedback for offline bandit learning of NMT parameters. We conduct a thorough analysis of the available explicit user judgments---five-star ratings of translation quality---and show that they are not reliable enough to yield significant improvements in bandit learning. In contr…
