We propose several ways of reusing subword embeddings and other weights in subword-aware neural language models. The proposed techniques do not benefit a competitive character-aware model, but some of them improve the performance of syllable- and morpheme-aware models while showing significant reductions in model sizes. We discover a simple hands-on principle: in a multi-layer input embedding model, layers should be tied consecutively bottom-up if reused at output. Our best morpheme-aware model with properly reused weights beats the competitive word-level model by a large margin across multiple languages and has 20%-87% fewer parameters.
This study addresses unsupervised subword modeling, i.e., learning feature representations that can distinguish subword units of a language. The proposed approach adopts a two-stage bottleneck feature (BNF) learning framework, consisting of autoregressive predictive coding (APC) as a front-end and a DNN-BNF model as a back-end. APC pretrained features are set as input features to a DNN-BNF model. A language-mismatched ASR system is used to provide cross-lingual phone labels for DNN-BNF model training. Finally, BNFs are extracted as the subword-discriminative feature representation. A second ai…
Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
Institute of Business Management（商业管理学院）；Dalhousie University（达尔豪斯大学）；Heidelberg University（海德堡大学）；Bahria University（巴里亚大学）；College of Computer Science and Engineering（计算机科学与工程学院）；Taibah University（塔伊巴大学）；Faculty of Computing and Information Technology（计算机与信息技术学院）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；School of Computer Science and Technology（计算机科学与技术学院）；Harbin Institute of Technology（哈尔滨工业大学）
This study addresses unsupervised subword modeling, i.e., learning acoustic feature representations that can distinguish between subword units of a language. We propose a two-stage learning framework that combines self-supervised learning and cross-lingual knowledge transfer. The framework consists of autoregressive predictive coding (APC) as the front-end and a cross-lingual deep neural network (DNN) as the back-end. Experiments on the ABX subword discriminability task conducted with the Libri-light and ZeroSpeech 2017 databases showed that our approach is competitive or superior to state-of-…
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi（印度理工学院德里分校传感器、仪器仪表和网络物理系统工程中心（SeNSE中心））；RSL Quantum, FITT, IIT Delhi（印度理工学院德里分校FITT的RSL量子公司）
iFLYTEK Research（科大讯飞研究）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；Harbin Institute of Technology（哈尔滨工业大学）；Research Center for Social Computing and Information Retrieval (SCIR)（社会计算与信息检索研究中心（SCIR））；iFLYTEK AI Research (Hebei)（科大讯飞AI研究（河北））；University of Maryland（马里兰大学）
Most pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations, by which OOV (out-of-vocab) words are almost avoidable. However, those methods split a word into subword units and make the representation incomplete and fragile. In this paper, we propose a character-aware pre-trained language model named CharBERT improving on the previous methods (such as BERT, RoBERTa) to tackle these problems. We first construct the contextual word embedding for each token from the sequential character representations, then fuse the repre…
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；College of Information Science and Technology and Artificial Intelligence, Nanjing Forestry University（南京林业大学信息科学技术与人工智能学院）
MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Technical University of Berlin（柏林工业大学）
