arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.09093cs.LGcs.AI
收藏

无需修改架构的精确比特级可逆Transformer研究

On Exact Bit-level Reversible Transformers Without Changing Architectures

University of Exeter(埃克塞特大学);NVIDIA(英伟达);Victoria University of Wellington(惠灵顿维多利亚大学)

Guoqiang Zhang, J. P. Lewis, W. B. Kleijn

AI总结:

本文提出BDIA-transformer,在不改动标准Transformer推理架构的前提下,结合双向积分近似与激活量化实现精确比特级可逆,既降低训练内存消耗,又通过集成正则化提升图像分类和翻译任务的验证性能。

摘要:

为降低训练过程中的内存消耗,学界已提出多种可逆深度神经网络(DNN)模型。但现有几乎所有可逆DNN要么需要特殊的非标准架构,要么需要对现有DNN架构进行大幅修改才能实现可逆性。本文提出BDIA-transformer,这是一种精确的比特级可逆Transformer,推理时采用的标准架构无需任何改动。其核心思路是:首先将每个Transformer块视为求解常微分方程(ODE)的欧拉积分近似,随后将双向积分近似(BDIA)技术融入神经架构,并结合激活量化,使其实现精确的比特级可逆性。训练过程中,我们让BDIA-transformer中的超参数$γ$在每个训练样本的每个Transformer块上随机取$\{0.5, -0.5\}$两个值之一,以对每两个连续的积分近似结果取平均。由此,BDIA-transformer可被视为训练一组由二元随机变量参数化的ODE求解器的集成,这一过程对模型起到正则化作用,进而提升验证准确率。前向传播过程中需要存储每个Transformer块的轻量辅助信息,用于补偿二值量化损失,以实现精确的比特级可逆性。推理阶段取期望$\mathbb{E}(γ)=0$,使得BDIA-transformer的最终架构除激活量化外,与普通Transformer完全一致。我们在图像分类和机器翻译两个任务上开展的实验表明,BDIA-transformer在验证性能上显著优于传统Transformer,同时所需训练内存大幅降低。

英文摘要:

Various reversible deep neural networks (DNN) models have been proposed to reduce memory consumption in the training process. However, almost all existing reversible DNNs either require special non-standard architectures or are constructed by modifying existing DNN architectures considerably to enable reversibility. In this work we present the BDIA-transformer, which is an exact bit-level reversible transformer that uses an unchanged standard architecture for inference. The basic idea is to first treat each transformer block as the Euler integration approximation for solving an ordinary differential equation (ODE) and then incorporate the technique of bidirectional integration approximation (BDIA) into the neural architecture, together with activation quantization to make it exactly bit-level reversible. In the training process, we let a hyper-parameter $γ$ in BDIA-transformer randomly take one of the two values $\{0.5, -0.5\}$ per training sample per transformer block for averaging every two consecutive integration approximations. As a result, BDIA-transformer can be viewed as training an ensemble of ODE solvers parameterized by a set of binary random variables, which regularizes the model and results in improved validation accuracy. Lightweight side information per transformer block is required to be stored in the forward process to account for binary quantization loss to enable exact bit-level reversibility. In the inference procedure, the expectation $\mathbb{E}(γ)=0$ is taken to make the resulting architectures of BDIA-transformer identical to transformers up to activation quantization. Our experiments in both image classification and language translation show that BDIA-transformers outperform their conventional counterparts significantly in terms of validation performance while also requiring considerably less training memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑