arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.02867cs.CL
收藏

通过homotokens训练语言模型导致过拟合延迟

Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling

Dalle Molle Institute for Artificial Intelligence (IDSIA)(达勒莫莱人工智能研究所);National University of Science and Technology POLITEHNICA Bucharest(科学与技术国家大学)

Adrian Cosma, Stefan Ruseti, Emilian Radoi, Mihai Dascalu

AI总结:

通过homotokens训练语言模型可延迟过拟合并提升泛化能力,方法通过辅助编码器和注意力机制实现tokenization不变性。

摘要:

子词分词在语言模型中引入了一个计算层,许多不同的token序列解码为相同的表面形式并保留意义,但会引发不同的内部计算。尽管这种非唯一性,语言模型通常使用单一的规范最长前缀分词进行训练。我们正式将homotokens(同一词项的不同有效子词分段)作为严格意义保持的数据增强形式。我们引入了一种轻量级的训练架构,通过辅助因果编码器和块因果交叉注意力,将规范的下一个token预测条件化于采样的homotoken变体,而无需修改训练目标或token接口。在数据受限的预训练中,homotoken增强在重复数据暴露下一致延迟过拟合,并在各种评估数据集上提高泛化能力。在多语言微调中,我们发现homotokens的有效性取决于分词器质量:当规范token高度压缩时收益最大,当分词器已过度碎片化输入时收益减少。总体而言,homotokens为语言模型诱导tokenization不变性提供了一种简单且模块化的机制。

英文摘要:

Subword tokenization introduces a computational layer in language models, where many distinct token sequences decode to the same surface form while preserving meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical deterministic tokenization. In this work, we use non-canonical subword variants as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled subword variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, augmentations with subword variants consistently delay overfitting under repeated data exposure and reduce held-out next-token loss across evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of training with auxiliary subwords depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑