ZeroSyl: 用于口语语言建模的简单零资源音节分词
ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling
AI总结：
提出ZeroSyl，一种无需训练的方法，直接从冻结的WavLM模型中提取音节边界和嵌入，实现竞争性的音节分割性能，并在词汇、句法和叙事基准上优于先前方法。
摘要：
纯语音语言模型旨在直接从原始音频中学习语言，无需文本资源。一个关键挑战是来自自监督语音编码器的离散标记会导致过长的序列，这促使了最近关于音节类单元的研究。然而，像Sylber和SyllableLM这样的方法依赖于复杂的多阶段训练流程。我们提出了ZeroSyl，一种简单的无需训练的方法，直接从冻结的WavLM模型中提取音节边界和嵌入。通过使用WavLM中间层特征的L2范数，ZeroSyl实现了具有竞争力的音节分割性能。得到的片段进行均值池化，使用K-means离散化，并用于训练语言模型。ZeroSyl在词汇、句法和叙事基准上优于先前的音节分词器。扩展实验表明，虽然更细粒度的单元有利于词汇任务，但我们发现的音节单元在句法建模方面表现出更好的扩展行为。
英文摘要：
Pure speech language models aim to learn language directly from raw audio without textual resources. A key challenge is that discrete tokens from self-supervised speech encoders result in excessively long sequences, motivating recent work on syllable-like units. However, methods like Sylber and SyllableLM rely on intricate multi-stage training pipelines. We propose ZeroSyl, a simple training-free method to extract syllable boundaries and embeddings directly from a frozen WavLM model. Using L2 norms of features in WavLM's intermediate layers, ZeroSyl achieves competitive syllable segmentation performance. The resulting segments are mean-pooled, discretized using K-means, and used to train a language model. ZeroSyl outperforms prior syllabic tokenizers across lexical, syntactic, and narrative benchmarks. Scaling experiments show that while finer-grained units are beneficial for lexical tasks, our discovered syllabic units exhibit better scaling behavior for syntactic modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
