BabyLM的首批词汇：作为音系探测任务的分词
BabyLM's First Words: Word Segmentation as a Phonological Probing Task
AI总结：
针对大语言模型音系分析难、非英语音系基准少的问题，本研究将分词作为音系探测任务，基于31种语言的儿童语音训练音位语言模型，提出提取词边界的无监督方法，验证了统计学习理论并为子词分词器训练提供新思路。
摘要：
语言模型为研究基于预测的语言学理论提供了关键框架，但利用大语言模型（LLM）开展音系分析难度较大：目前英语之外的音系基准数量很少，且LLM采用的标准输入表示（字形子词）并不适合分析音位表征。在本研究中，我们论证了分词可作为音系探测任务，借此研究在面向儿童的语音语料上训练的、基于音位的语言模型在31种语言中学习到的表征。我们参照分词计算模型，利用“词首位置预测误差达到峰值”这一观测结果，提出了从训练好的模型中提取词边界的无监督方法。我们还通过线性探测发现，即便训练数据中没有词边界标注，这些模型也能隐式追踪词边界。这项跨语言研究验证了语言习得的统计学习理论，也为训练子词分词器的新方法提供了实证依据。
英文摘要：
Language models provide a key framework for studying linguistic theories based on prediction, but phonological analysis using large language models (LLMs) is difficult; there are few phonological benchmarks beyond English and the standard input representation used in LLMs (subwords of graphemes) is not suitable for analyzing the representation of phonemes. In this work, we demonstrate how word segmentation can be used as a phonological probing task, allowing us to study the representations learned by phoneme-based language models trained on child-directed speech across 31 languages. Following computational models of word segmentation, we present unsupervised methods for extracting word boundaries from a trained model using the observation that prediction-error peaks at the start of words. We also use linear probes to identify that these models implicitly track word boundaries, even when they do not appear in training. This cross-lingual work corroborates statistical learning theories of acquisition and empirically motivates new methods for training subword tokenizers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
