arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04012cs.NEcs.AIcs.LGeess.SPmath.STstat.TH
收藏

量化近似正交循环神经网络

On Orthogonal Recurrent Neural Networks Quantization

Institut de Mathématiques de Toulouse(图卢兹数学研究所);Université de Toulouse(图卢兹大学);CNRS(法国国家科学研究中心);UPS IMT;Institut de Recherche Technologique Saint Exupéry(圣埃克苏佩里技术研究院)

Armand Foucault, Franck Mamalet, François Malgouyres

AI总结:

本文针对正交循环神经网络难以部署在紧凑设备的问题,提出量化近似正交循环神经网络,通过结合量化感知训练与正交投影学习权重,并对激活值进行训练后量化,在4比特量化下仍能取得与全精度模型相近的性能。

摘要:

近年来,正交循环神经网络(ORNNs)因其能够处理涉及长期依赖的任务(如复制任务)且具有线性复杂度而受到关注。然而,现有的ORNN使用全精度权重和激活值,这阻碍了它们在紧凑型设备上的部署。本文探索了ORNN中权重矩阵的量化,提出了量化近似正交循环神经网络(QORNNs)。此类网络的构建一直是一个悬而未决的问题,其固有的不稳定性已得到公认。我们提出并研究了两种学习QORNN的策略,将量化感知训练(QAT)与正交投影相结合。我们还研究了激活值的训练后量化,以实现循环回路的纯整数计算。在多种标准基准上,即使采用4比特量化,最高效的模型也能取得与最先进的全精度ORNN、LSTM和FastRNN相近的结果。

英文摘要:

In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the Copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices. In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient fully-quantized models are currently the only alternatives for solving the Copy-task with 1000 timesteps. They also surpass existing quantized models on tasks requiring long-term dependencies, such as permuted MNIST as well as some Internet of Things (IoT) tasks. However, they are less effective on tasks where long short-term memory (LSTMs) and other gated RNNs outperform ORNNs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑