arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.15425cs.LGcs.AIcs.CLstat.ML
收藏

自注意力神经网络的经验容量模型

Empirical Capacity Model for Self-Attention Neural Networks

Maastricht University(马斯特里赫特大学)

Aki Härmä, Marcin Pietrasik, Anna Wilbik

AI总结:

本文通过常见训练算法与合成数据研究Transformer的记忆容量,提出经验容量模型ECM,用于在给定目标记忆能力时设计参数数量最优的任务专用Transformer。

摘要:

大型预训练自注意力神经网络(即Transformer)近来在各类任务中取得了巨大成功。模型在给定任务上的性能取决于其记忆和泛化训练数据的能力。拥有数十亿参数的大型Transformer模型在理论上具有巨大的内容记忆容量。然而,当前的优化算法远未达到理论容量,且容量高度依赖于具体内容。在本文中,我们关注使用常见训练算法和合成训练数据所获得的这些模型的记忆容量。基于实验结果,我们为通用Transformer推导出一个经验容量模型(ECM)。在任务的目标记忆能力可以被定义的情况下,该ECM可用于设计具有最优参数数量的任务专用Transformer模型。

英文摘要:

Large pretrained self-attention neural networks, or transformers, have been very successful in various tasks recently. The performance of a model on a given task depends on its ability to memorize and generalize the training data. Large transformer models, which may have billions of parameters, in theory have a huge capacity to memorize content. However, the current algorithms for the optimization fall short of the theoretical capacity, and the capacity is also highly dependent on the content. In this paper, we focus on the memory capacity of these models obtained using common training algorithms and synthetic training data. Based on the results, we derive an empirical capacity model (ECM) for a generic transformer. The ECM can be used to design task-specific transformer models with an optimal number of parameters in cases where the target memorization capability of the task can be defined.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑