ProFormer：基于设备上LSH投影的Transformers
ProFormer: Towards On-Device LSH Projection Based Transformers
摘要：
基于文本的神经模型的核心是单词表示，它功能强大，但占用大量内存，这使得部署到具有内存限制的设备（如手机，手表和物联网）具有挑战性。为了克服这些挑战，我们引入了ProFormer --一种基于投影的Transformer架构，它更快、更轻，适合部署到内存受限的设备上，并保护用户隐私。我们使用LSH投影层动态地动态生成单词表示，而不嵌入查找表，从而将内存占用从O（V.d）显著减少到O（T），其中V是词汇大小，d是嵌入维度大小，T是LSH投影表示的维度。我们还提出了一个局部投影注意（LPA）层，它使用自注意将N个LSH单词投影的输入序列转换为N/K个表示的序列，从而使计算量二次减少O（K^2）。我们评估了ProFormer在多个文本分类任务上的表现，并观察到其在短文本分类方面优于现有最先进的设备上方法，而在长文本分类任务上的表现则与之相当。与2层BERT模型相比，ProFormer将嵌入内存占用从92.16 MB减少到1.3 KB，计算开销减少了16倍，这是非常令人印象深刻的，使其成为最快和最小的设备模型。
英文摘要：
At the heart of text based neural models lay word representations, which are powerful but occupy a lot of memory making it challenging to deploy to devices with memory constraints such as mobile phones, watches and IoT. To surmount these challenges, we introduce ProFormer -- a projection based transformer architecture that is faster and lighter making it suitable to deploy to memory constraint devices and preserve user privacy. We use LSH projection layer to dynamically generate word representations on-the-fly without embedding lookup tables leading to significant memory footprint reduction from O(V.d) to O(T), where V is the vocabulary size, d is the embedding dimension size and T is the dimension of the LSH projection representation. We also propose a local projection attention (LPA) layer, which uses self-attention to transform the input sequence of N LSH word projections into a sequence of N/K representations reducing the computations quadratically by O(K^2). We evaluate ProFormer on multiple text classification tasks and observed improvements over prior state-of-the-art on-device approaches for short text classification and comparable performance for long text classification tasks. In comparison with a 2-layer BERT model, ProFormer reduced the embedding memory footprint from 92.16 MB to 1.3 KB and requires 16 times less computation overhead, which is very impressive making it the fastest and smallest on-device model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
