The success of pretrained transformer language models (LMs) in natural language processing has led to a wide range of pretraining setups. In particular, these models employ a variety of subword tokenization methods, most notably byte-pair encoding (BPE) (Sennrich et al., 2016; Gage, 1994), the WordPiece method (Schuster and Nakajima, 2012), and unigram language modeling (Kudo, 2018), to segment text. However, to the best of our knowledge, the literature does not contain a direct evaluation of the impact of tokenization on language model pretraining. We analyze differences between BPE and unigram LM tokenization, finding that the latter method recovers subword units that align more closely with morphology and avoids problems stemming from BPE's greedy construction procedure. We then compare the fine-tuned task performance of identical transformer masked language models pretrained with these tokenizations. Across downstream tasks and two languages (English and Japanese), we find that the unigram LM tokenization method matches or outperforms BPE. We hope that developers of future pretrained LMs will consider adopting the unigram LM method over the more prevalent BPE.
Contextual morphologically-guided tokenization for Latin encoder models
Manning College of Information & Computer Sciences, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校信息与计算机科学学院）；Institute for the Study of the Ancient World, New York University（纽约大学古代世界研究所）
International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；Hong Kong University of Science and Technology（香港科技大学）；Yale University（耶鲁大学）；National University of Singapore（新加坡国立大学）
The zero-shot cross-lingual ability of models pretrained on multilingual and even monolingual corpora has spurred many hypotheses to explain this intriguing empirical result. However, due to the costs of pretraining, most research uses public models whose pretraining methodology, such as the choice of tokenization, corpus size, and computational budget, might differ drastically. When researchers pretrain their own models, they often do so under a constrained budget, and the resulting models might underperform significantly compared to SOTA models. These experimental differences led to various …
Neural rankers based on deep pretrained language models (LMs) have been shown to improve many information retrieval benchmarks. However, these methods are affected by their the correlation between pretraining domain and target domain and rely on massive fine-tuning relevance labels. Directly applying pretraining methods to specific domains may result in suboptimal search quality because specific domains may have domain adaption problems, such as the COVID domain. This paper presents a search system to alleviate the special domain adaption problem. The system utilizes the domain-adaptive pretra…
Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）
Effectively finetuning pretrained language models (PLMs) is critical for their success in downstream tasks. However, PLMs may have risks in overfitting the pretraining tasks and data, which usually have gap with the target downstream tasks. Such gap may be difficult for existing PLM finetuning methods to overcome and lead to suboptimal performance. In this paper, we propose a very simple yet effective method named NoisyTune to help better finetune PLMs on downstream tasks by adding some noise to the parameters of PLMs before fine-tuning. More specifically, we propose a matrix-wise perturbing m…
A wide range of NLP tasks benefit from the fine-tuning of pretrained language models (PLMs). However, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine-tuned model. We consider the gap between pretraining and downstream tasks hinders the training of these redundant parameters, and results in a suboptimal performance of the overall model. In this paper, we present PATS (Perturbation According To Sensitivity), a noisy training mechanism which considers each parameter's importance in the downstream task to help fine-tune PLMs. The main i…
MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Technical University of Berlin（柏林工业大学）
School of Information Science and Technology, Vidyasirimedhi Institution of Science and Technology（威集玛迪科学技术大学信息科学技术学院）；PyThaiNLP
Transformer-based language models, more specifically BERT-based architectures have achieved state-of-the-art performance in many downstream tasks. However, for a relatively low-resource language such as Thai, the choices of models are limited to training a BERT-based model based on a much smaller dataset or finetuning multi-lingual models, both of which yield suboptimal downstream performance. Moreover, large-scale multi-lingual pretraining does not take into account language-specific features for Thai. To overcome these limitations, we pretrain a language model based on RoBERTa-base architect…
ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
NII LLMC；Osaka University（大阪大学）；Nara Institute of Science and Technology（奈良科学技術大學）；University College London（伦敦大学学院）；Kyoto University（京都大学）；Waseda University（早稻田大学）
Stanford University（斯坦福大学）；Amazon（亚马逊）；AWS AI HLT（AWS 人工智能人类语言技术）
Contextual embedding-based language models trained on large data sets, such as BERT and RoBERTa, provide strong performance across a wide range of tasks and are ubiquitous in modern NLP. It has been observed that fine-tuning these models on tasks involving data from domains different from that on which they were pretrained can lead to suboptimal performance. Recent work has explored approaches to adapt pretrained language models to new domains by incorporating additional pretraining using domain-specific corpora and task data. We propose an alternative approach for transferring pretrained lang…
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Computational Story Lab（计算故事实验室）；Vermont Complex Systems Institute（佛蒙特复杂系统研究所）；University of Vermont（佛蒙特大学）；Department of Computer Science（计算机科学系）；St. Michael’s College（圣米歇尔学院）；Communication Media and Marginalization Lab（传播媒体与边缘化实验室）；Northeastern University（东北大学）；Department of Mathematics & Statistics（数学与统计学系）
Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；IIIT Hyderabad（海得拉巴国际信息技术学院）
Vision-and-language navigation (VLN) aims to enable embodied agents to navigate in realistic environments using natural language instructions. Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging. Recent methods explore pretraining to improve generalization, however, the use of generic image-caption datasets or existing small-scale VLN environments is suboptimal and results in limited improvements. In this work, we introduce BnB, a large-scale and diverse in-domain VL…
Data Engineering for Scaling Language Models to 128K Context
University of Edinburgh（爱丁堡大学）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；University of Melbourne（墨尔本大学）；Ohio State University（俄亥俄州立大学）；University of Washington（华盛顿大学）；MIT（麻省理工学院）；UIUC（伊利诺伊大学香槟分校）
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Microsoft（微软）；DeepMind；Allen Institute for AI（艾伦人工智能研究所）
Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's complexity scales quadratically with sequence length. Efficient transformer variants have received increasing interest in recent works. Among them, a linear-complexity recurrent variant has proven well suited for autoregressive generation. It approximates the softmax attention with randomized or heuristic feature maps, but can be difficult to train and may yield suboptimal accuracy. This work aims to convert a pretrai…
