arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.12893cs.LGcs.AI
收藏

混合动态剪枝:高效Transformer推理的路径

Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference

Ghadeer Jaradat, Mohammed Tolba, Ghada Alsuhli, Hani Saleh, Mahmoud Al-Qutayri, Thanos Stouraitis, Baker Mohammad

AI总结:

针对Transformer推理计算和内存开销大的问题,提出混合动态剪枝(HDP)算法-架构协同设计,利用头部与块稀疏性及近似方法加速,并配套协处理器架构。

摘要:

在深度学习领域,Transformer模型已变得非常重要,推动了从语言理解到图像识别等众多领域的进步,涵盖了广泛的应用。尽管取得了成功,但这些模型在实时应用中的部署,尤其是在边缘设备上,由于其二次方计算强度和内存需求,带来了重大挑战。为克服这些挑战,我们提出了一种新颖的混合动态剪枝(HDP),这是一种高效的算法-架构协同设计方法,利用头部稀疏性、块稀疏性和近似机会来减少注意力中的计算量并减少内存访问。基于对注意力分数和注意力头中巨大冗余的观察,我们提出了一种新颖的基于整数的行平衡块剪枝,在运行时剪除注意力矩阵中不重要的块,同时提出基于整数的头部剪枝,在运行时早期阶段检测并剪除不重要的头部。此外,我们提出了一种近似方法来减少注意力计算。为了以更低延迟和更高能效高效支持这些方法,我们提出了一种HDP协处理器架构。

英文摘要:

In the world of deep learning, Transformer models have become very significant, leading to improvements in many areas from understanding language to recognizing images, covering a wide range of applications. Despite their success, the deployment of these models in real-time applications, particularly on edge devices, poses significant challenges due to their quadratic computational intensity and memory demands. To overcome these challenges we introduce a novel Hybrid Dynamic Pruning (HDP), an efficient algorithm-architecture co-design approach that accelerates transformers using head sparsity, block sparsity and approximation opportunities to reduce computations in attention and reduce memory access. With the observation of the huge redundancy in attention scores and attention heads, we propose a novel integer-based row-balanced block pruning to prune unimportant blocks in the attention matrix at run time, also propose integer-based head pruning to detect and prune unimportant heads at an early stage at run time. Also we propose an approximation method that reduces attention computations. To efficiently support these methods with lower latency and power efficiency, we propose a HDP co-processor architecture.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑