arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.22906cs.CL
收藏

从咿呀语到单词:在连续音素流上预训练语言模型

From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes

University of Cambridge(剑桥大学);ALTA Institute, University of Cambridge(剑桥大学ALTA研究所);University of Göttingen(哥廷根大学)

Zébulon Goriely, Richard Diehl Martinez, Andrew Caines, Lisa Beinborn, Paula Buttery

AI总结:

针对语言模型音素训练效果难以评估的问题,研究提出文本转连续音素流流水线,在BabyLM数据集及标准基准上验证,发现音素训练略降传统任务性能但具分析与实用价值。

摘要:

语言模型通常以默认正字法形式在大规模文本语料上训练,但这并非唯一选择;将数据表示为音素流可带来独特优势,从对音系语言习得的更深入洞见,到提升基于语音的任务性能。挑战在于评估基于音素的训练的影响,因为大多数基准也都是正字法形式的。为解决此问题,我们开发了一套将文本数据集转换为连续音素流的流水线。我们将该流水线应用于BabyLM挑战赛的1亿词预训练数据集,以及标准语言和语法基准,从而能够使用音素输入表示来预训练并评估模型。我们的结果表明,尽管基于音素的训练会略微降低传统语言理解任务的性能,但它具备宝贵的分析与实用价值。

英文摘要:

Language models are typically trained on large corpora of text in their default orthographic form. However, this is not the only option; representing data as streams of phonemes can offer unique advantages, from deeper insights into phonological language acquisition to improved performance on sound-based tasks. The challenge lies in evaluating the impact of phoneme-based training, as most benchmarks are also orthographic. To address this, we develop a pipeline to convert text datasets into a continuous stream of phonemes. We apply this pipeline to the 100-million-word pre-training dataset from the BabyLM challenge, as well as to standard language and grammatical benchmarks, enabling us to pre-train and evaluate a model using phonemic input representations. Our results show that while phoneme-based training slightly reduces performance on traditional language understanding tasks, it offers valuable analytical and practical benefits.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑