SPARTAN：用于参数高效Transformers的稀疏层次存储器
SPARTAN: Sparse Hierarchical Memory for Parameter-Efficient Transformers
摘要：
微调预训练的语言模型（PLM）在一系列下游任务上实现了令人印象深刻的性能，因此它们的规模越来越大。由于每个任务都需要模型的不同副本，因此这种范例对于移动电话等存储受限的边缘设备是不可行的。在本文中，我们提出了SPARTAN，一个参数高效（PE）和计算速度快的边缘设备，每个Transformer层后增加分层组织的稀疏内存架构。SPARTAN冻结PLM参数并仅微调其内存，从而通过为不同任务重复使用PLM主干来显著降低存储成本。SPARTAN包含两个级别的内存，在第一级中为每个输入仅选择父节点的稀疏子集，并且与这些父节点对应的子单元用于计算输出表示。与PE基线（适配器）相比，这种稀疏性与其他架构优化相结合，在Raspberry Pi 4上进行推理时，SPARTAN的吞吐量提高了90%以上，同时在GLUE基准测试中也超过后者0.1个点。此外，它可以在Few-Shot设置中训练速度快34%，同时在适配器的0.9点内执行。定性分析表明，SPARTAN中不同的父细胞专注于不同的主题，从而有效地划分责任。
英文摘要：
Fine-tuning pre-trained language models (PLMs) achieves impressive performance on a range of downstream tasks, and their sizes have consequently been getting bigger. Since a different copy of the model is required for each task, this paradigm is infeasible for storage-constrained edge devices like mobile phones. In this paper, we propose SPARTAN, a parameter efficient (PE) and computationally fast architecture for edge devices that adds hierarchically organized sparse memory after each Transformer layer. SPARTAN freezes the PLM parameters and fine-tunes only its memory, thus significantly reducing storage costs by re-using the PLM backbone for different tasks. SPARTAN contains two levels of memory, with only a sparse subset of parents being chosen in the first level for each input, and children cells corresponding to those parents being used to compute an output representation. This sparsity combined with other architecture optimizations improves SPARTAN's throughput by over 90% during inference on a Raspberry Pi 4 when compared to PE baselines (adapters) while also outperforming the latter by 0.1 points on the GLUE benchmark. Further, it can be trained 34% faster in a few-shot setting, while performing within 0.9 points of adapters. Qualitative analysis shows that different parent cells in SPARTAN specialize in different topics, thus dividing responsibility efficiently.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
