Language model pre-training based on large corpora has achieved tremendous success in terms of constructing enriched contextual representations and has led to significant performance gains on a diverse range of Natural Language Understanding (NLU) tasks. Despite the success, most current pre-trained language models, such as BERT, are trained based on single-grained tokenization, usually with fine-grained characters or sub-words, making it hard for them to learn the precise meaning of coarse-grained words and phrases. In this paper, we propose a simple yet effective pre-training method named LICHEE to efficiently incorporate multi-grained information of input text. Our method can be applied to various pre-trained language models and improve their representation capability. Extensive experiments conducted on CLUE and SuperGLUE demonstrate that our method achieves comprehensive improvements on a wide variety of NLU tasks in both Chinese and English with little extra inference cost incurred, and that our best ensemble model achieves the state-of-the-art performance on CLUE benchmark competition.
FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs
Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国）；Amazon GenAI（亚马逊生成人工智能）；Qatar Computing Research Institute（卡塔尔计算研究所）；University of Virginia（弗吉尼亚大学）
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，北京，中国）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院，北京，中国）
Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models
The University of Tokyo（东京大学）；Keio University（.keio大学）；Graduate School of Information Science and Technology（信息科学与技术研究生院）；Faculty of Science and Technology（科学与技术学部）
M2Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models
The Hong Kong Polytechnic University（香港理工大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology（香港科技大学）
