通过homotokens训练语言模型导致过拟合延迟
Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling
AI总结:
通过homotokens训练语言模型可延迟过拟合并提升泛化能力,方法通过辅助编码器和注意力机制实现tokenization不变性。
摘要:
子词分词在语言模型中引入了一个计算层,许多不同的token序列解码为相同的表面形式并保留意义,但会引发不同的内部计算。尽管这种非唯一性,语言模型通常使用单一的规范最长前缀分词进行训练。我们正式将homotokens(同一词项的不同有效子词分段)作为严格意义保持的数据增强形式。我们引入了一种轻量级的训练架构,通过辅助因果编码器和块因果交叉注意力,将规范的下一个token预测条件化于采样的homotoken变体,而无需修改训练目标或token接口。在数据受限的预训练中,homotoken增强在重复数据暴露下一致延迟过拟合,并在各种评估数据集上提高泛化能力。在多语言微调中,我们发现homotokens的有效性取决于分词器质量:当规范token高度压缩时收益最大,当分词器已过度碎片化输入时收益减少。总体而言,homotokens为语言模型诱导tokenization不变性提供了一种简单且模块化的机制。
英文摘要:
Subword tokenization introduces a computational layer in language models, where many distinct token sequences decode to the same surface form while preserving meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical deterministic tokenization. In this work, we use non-canonical subword variants as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled subword variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, augmentations with subword variants consistently delay overfitting under repeated data exposure and reduce held-out next-token loss across evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of training with auxiliary subwords depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。