Language models require tokenized inputs. However, tokenization strategies for continuous data like audio and vision are often based on simple heuristics such as fixed sized convolutions or discrete clustering, which do not necessarily align with the semantic structure of the data. For speech in particular, the high resolution of waveforms (16,000 samples/second or more) presents a significant challenge as speech-based language models have had to use several times more tokens per word than text-based language models. In this work, we introduce a controllable self-supervised technique to merge speech representations into coarser syllable-like units while still preserving semantic information. We do this by 1) extracting noisy boundaries through analyzing correlations in pretrained encoder losses and 2) iteratively improving model representations with a novel distillation technique. Our method produces controllable-rate semantic units at as low as 5Hz and 60bps and achieves SotA in syllabic segmentation and clustering. Using these coarse tokens, we successfully train SyllableLM, a Speech Language Model (SpeechLM) that matches or outperforms current SotA SpeechLMs on a range of spoken language modeling tasks. SyllableLM also achieves significant improvements in efficiency with a 30x reduction in training compute and a 4x wall-clock inference speedup.
Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models
The University of Tokyo（东京大学）；Keio University（.keio大学）；Graduate School of Information Science and Technology（信息科学与技术研究生院）；Faculty of Science and Technology（科学与技术学部）
A Joint Spectro-Temporal Relational Thinking Based Acoustic Modeling Framework
School of Electrical Engineering and Telecommunications, University of New South Wales（新南威尔士大学电气与通信工程学院）；School of Computing and Information Systems, University of Melbourne（墨尔本大学计算与信息系统学院）
BrainStratify: Coarse-to-Fine Disentanglement of Intracranial Neural Dynamics
Beijing Normal University（北京师范大学）；Peking University（北京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Chinese Institute for Brain Research（中国脑科学研究院）；Capital Medical University, Xuanwu Hospital, Beijing（首都医科大学宣武医院）
Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translation
Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University（天津大学认知计算与应用天津市重点实验室）；Nanyang Technological University（南洋理工大学）；Huiyan Technology Company, Ltd.（慧眼科技有限公司）；Chinese Academy of Sciences（中国科学院）；Tibet University（西藏大学）
Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models
Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
Cognitive Computational Neuroscience Group（认知计算神经科学组）；Friedrich-Alexander-Universität Erlangen–Nürnberg (FAU)（弗赖堡-亚历山大-大学埃尔兰根-纽伦堡（FAU））；Mannheim Center for Neuromodulation and Neuroprosthetics (MCNN)（曼海姆神经调制与神经假体中心（MCNN））；University Hospital Mannheim, University Heidelberg（曼海姆大学医院，海德堡大学）
