arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.08412cs.CLcs.AR
收藏

矢量-矢量-矩阵结构:一种新的硬件感知NLP低延迟推理框架

Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications

Lightelligence, Inc.(光智公司);Massachusetts Institute of Technology(麻省理工学院);Qatar Computing Research Institute, HBKU(卡塔尔计算研究所(哈马德·本·哈利法大学))

Matthew Khoury, Rumen Dangovski, Longwu Ou, Preslav Nakov, Yichen Shen, Li Jing

摘要:

深度神经网络已成为构建可靠的自然语言处理(NLP)应用程序的标准方法,从神经机器翻译(NMT)到对话系统。然而,通过增加模型大小来提高精度需要大量的硬件计算,这可能会在推理时显着降低NLP应用程序的速度。为了解决这个问题,我们提出了一种新的向量-向量-矩阵架构(VVMA),它大大减少了NMT推理时间的延迟。此架构利用具有低延迟向量-向量运算和较高延迟向量-矩阵运算的专用硬件。它还减少了几乎所有依赖高效矩阵乘法器的模型的参数和FLOP数量,而不会显著影响精度。我们目前的实证结果表明,我们的框架可以减少延迟的序列到序列和Transformer模型用于NMT的四个因素。最后,我们有证据表明,我们的VVMA扩展到其他领域,我们讨论了新的硬件,其有效的使用。

英文摘要:

Deep neural networks have become the standard approach to building reliable Natural Language Processing (NLP) applications, ranging from Neural Machine Translation (NMT) to dialogue systems. However, improving accuracy by increasing the model size requires a large number of hardware computations, which can slow down NLP applications significantly at inference time. To address this issue, we propose a novel vector-vector-matrix architecture (VVMA), which greatly reduces the latency at inference time for NMT. This architecture takes advantage of specialized hardware that has low-latency vector-vector operations and higher-latency vector-matrix operations. It also reduces the number of parameters and FLOPs for virtually all models that rely on efficient matrix multipliers without significantly impacting accuracy. We present empirical results suggesting that our framework can reduce the latency of sequence-to-sequence and Transformer models used for NMT by a factor of four. Finally, we show evidence suggesting that our VVMA extends to other domains, and we discuss novel hardware for its efficient use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑