arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.11985cs.CLcs.AIcs.LG
收藏

公平语言模型悖论

The Fair Language Model Paradox

MIT(麻省理工学院);ETH Zurich(苏黎世联邦理工学院);Texas A&M University(德克萨斯农工大学);Brown University(布朗大学)

Andrea Pinto, Tomer Galanti, Randall Balestriero

AI总结:

本文揭示了大型语言模型训练中权重衰减在token级别引入的性能偏差,发现低频token随权重衰减增加而遭受不成比例的贬值,呼吁开发新正则化技术以确保所有token的公平性。

摘要:

大型语言模型(LLMs)被广泛应用于实际场景,但其在 token 级别的训练动态却鲜为人知。评估通常依赖于在批次级别测量的聚合训练损失,这忽略了由(i)不同的 token 级别动态和(ii)超参数引入的结构性偏差所产生的细微逐 token 偏差。虽然权重衰减常用于稳定训练,但我们发现它悄然引入了仅在 token 级别才能检测到的性能偏差。事实上,我们在不同数据集规模、模型架构以及规模从 270M 到 3B 参数的模型上实证表明,随着权重衰减增加,低频 token 遭到了不成比例的贬值。这尤其令人担忧,因为这些被忽视的低频 token 在大多数语言的 token 分布中占绝大多数,亟需开发新的正则化技术来确保所有可用 token 的公平性。

英文摘要:

Large Language Models (LLMs) are widely deployed in real-world applications, yet little is known about their training dynamics at the token level. Evaluation typically relies on aggregated training loss, measured at the batch level, which overlooks subtle per-token biases arising from (i) varying token-level dynamics and (ii) structural biases introduced by hyperparameters. While weight decay is commonly used to stabilize training, we reveal that it silently introduces performance biases detectable only at the token level. In fact, we empirically show across different dataset sizes, model architectures and sizes ranging from 270M to 3B parameters that as weight decay increases, low-frequency tokens are disproportionately depreciated. This is particularly concerning, as these neglected low-frequency tokens represent the vast majority of the token distribution in most languages, calling for novel regularization techniques that ensure fairness across all available tokens.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑