The maximum element of the vector output by the Softmax function approaches zero as the input vector size increases. Transformer-based language models rely on Softmax to compute attention scores, causing the attention distribution to flatten as the context size grows. This reduces the model's ability to prioritize key information effectively and potentially limits its length generalization. To address this problem, we propose Scalable-Softmax (SSMax), which replaces Softmax in scenarios where the input vector size varies. SSMax can be seamlessly integrated into existing Transformer-based architectures. Experimental results in language modeling show that models using SSMax not only achieve faster loss reduction during pretraining but also significantly improve performance in long contexts and key information retrieval. Furthermore, an analysis of attention scores reveals that SSMax enables the model to focus attention on key information even in long contexts. Additionally, although models that use SSMax from the beginning of pretraining achieve better length generalization, those that have already started pretraining can still gain some of this ability by replacing Softmax in the attention layers with SSMax, either during or after pretraining.
HiMA: A Fast and Scalable History-based Memory Access Engine for Differentiable Neural Computer
University of Michigan(密歇根大学)
记忆增强神经网络(MANN)在外部记忆的帮助下,在许多任务中提供更好的推理性能。最近开发的可微神经计算机(DNC)是一种MANN,已被证明在表示复杂的数据结构和学习长期依赖关系方面表现出色。DNC的更高的性能来自于新的基于历史的注意力机制,除了以前使用的基于内容的注意力机制。基于历史的机制需要各种新的计算原语和状态存储器,这是不支持现有的神经网络(NN)或MANN加速器。我们提出了HiMA,一个平铺的,基于历史的内存访问引擎与分布式存储器在瓷砖。HiMA采用多模式片上网络(NoC)来减少通信延迟并提高可扩展性。一个最佳的子矩阵明智的内存分区策略,以减少的NoC流量的量,和一个两阶段的使用排序方法,利用分布式瓦片,以提高计算速度。为了使HiMA从根本上可扩展,我们创建了一个分布式版本的DNC,称为DNC-D,允许几乎所有的内存操作都应用于本地内存,并通过可训练的加权求和来产生全局内存输出。两个近似技术,使用撇取和softmax近似,提出了进一步提高硬件效率。HiMA原型在RTL中创建,并在40 nm技术中合成。通过模拟,HiMA运行DNC和DNC-D的速度比最先进的MANN加速器高6.47倍和39.1倍,面积效率高22.8倍和164.3倍,能源效率高6.1倍和61.2倍。与Nvidia 3080 Ti GPU相比,HiMA在运行DNC和DNC-D时的加速分别高达437倍和2,646倍…
Instituto Superior Técnico, Universidade de Lisboa(里斯本大学技术高级学院);Instituto de Telecomunicações(电信研究所);TransPerfect(TransPerfect公司);ELLIS Unit Lisbon(里斯本ELLIS单位)