Motivated by the fact that most of the information relevant to the prediction of target tokens is drawn from the source sentence $S=s_1, \ldots, s_S$, we propose truncating the target-side window used for computing self-attention by making an $N$-gram assumption. Experiments on WMT EnDe and EnFr data sets show that the $N$-gram masked self-attention model loses very little in BLEU score for $N$ values in the range $4, \ldots, 8$, depending on the task.
Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT's Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of n tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained information into pre-train…
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative decoding, where we first predict all of the target words non-autoregressively, and then repeatedly mask out and regenerate the subset of words that the model is least confident about. By applying this strategy for a constant number of iterations, our model improves state-of-the-…
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）
In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC). SAD optimizes the online inference efficiency for GEC by two innovations: 1) it aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism; 2) it uses a shallow decoder instead of the conventional Transformer architecture with balanced encoder-decoder depth to reduce the computational cost during inference. Experiments in both En…
Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models
Zhejiang University（浙江大学）；RIKEN Center for Advanced Intelligence Project（日本理化学研究院先进智能项目中心）；The Institute of Statistical Mathematics（统计数学研究所）；Agency for Science, Technology and Research (A ⋆ \star STAR)（科技研究局（A ⋆ STAR））
Accurate terminology translation is crucial for ensuring the practicality and reliability of neural machine translation (NMT) systems. To address this, lexically constrained NMT explores various methods to ensure pre-specified words and phrases appear in the translation output. However, in many cases, those methods are studied on general domain corpora, where the terms are mostly uni- and bi-grams (>98%). In this paper, we instead tackle a more challenging setup consisting of domain-specific corpora with much longer n-gram and highly specialized terms. Inspired by the recent success of masked …
Attention Meets Reachability: Structural Equivalence and Efficiency in Grammar-Constrained LLM Decoding
Department of Computer Engineering, Bahçeşehir University, Istanbul, Turkey（计算机工程系，巴切希尔大学，伊斯坦布尔，土耳其）；Department of Industrial Engineering, Bahçeşehir University, Istanbul, Turkey（工业工程系，巴切希尔大学，伊斯坦布尔，土耳其）
Stability-Weighted Decoding for Diffusion Language Models
Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，Location，Country）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，Location，Country）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, China（数据科学与人工智能系，香港理工大学，香港，中国）
Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow
Zhejiang University（浙江大学）；Ant Group（蚂蚁集团）；Shanghai Jiao Tong University（上海交通大学）；University of Chinese Academy of Social Sciences（中国社会科学院大学）；Westlake University（西湖大学）
AADNet: Exploring EEG Spatiotemporal Information for Fast and Accurate Orientation and Timbre Detection of Auditory Attention Based on A Cue-Masked Paradigm
West China Hospital, Sichuan University（四川大学华西医院）；Institute of Electronic Engineering, China Academy of Engineering Physics（中国工程物理研究院电子工程研究所）
One-step lowest-variance selection in a Gaussian random-field model motivated by masked diffusion: Total correlation and a square root collision threshold
