MERGE:快速隐私文本生成
MERGE: Fast Private Text Generation
AI总结:
针对两方隐私保护方法难以支持自回归 NLG 且训练收敛、推理效率不足的问题,本文提出 MERGE 框架,通过复用隐藏状态和重组线性运算实现高效隐私文本生成。
摘要:
语言模型参数量的急剧增加引发了在云服务器上部署模型的新趋势,也使基于 Transformer 的模型的隐私推理受到越来越多关注。然而,现有的两方隐私保护技术只考虑了自然语言理解(NLU)场景。自然语言生成(NLG)中的隐私推理对翻译和代码补全等应用至关重要,却仍未得到充分研究。此外,以往的隐私保护技术在模型训练期间存在收敛问题,并且在用于 NLG 模型时推理速度较差,原因是它们忽视了自回归生成中耗时的操作。为解决这些问题,我们提出了一个面向基于 Transformer 的语言模型的快速隐私文本生成框架,即 MERGE。MERGE 将输出隐藏状态复用为词嵌入,以绕过嵌入计算,并重新组织 Transformer 模块中的线性运算来加速前向过程。大量实验表明,在序列长度为 512 时,MERGE 相比普通加密模型实现了 26.5 倍加速,并降低 80% 的通信成本,同时相比最先进的近似模型最高实现 10 倍加速。
英文摘要:
The drastic increase in language models' parameters has led to a new trend of deploying models in cloud servers, raising growing concerns about private inference for Transformer-based models. Existing two-party privacy-preserving techniques, however, only take into account natural language understanding (NLU) scenarios. Private inference in natural language generation (NLG), crucial for applications like translation and code completion, remains underexplored.In addition, previous privacy-preserving techniques suffer from convergence issues during model training and exhibit poor inference speed when used with NLG models due to the neglect of time-consuming operations in auto-regressive generations. To address these issues, we propose a fast private text generation framework for Transformer-based language models, namely MERGE.MERGE reuses the output hidden state as the word embedding to bypass the embedding computation and reorganize the linear operations in the Transformer module to accelerate the forward procedure. Extensive experiments show that MERGE achieves a 26.5x speedup to the vanilla encrypted model under the sequence length 512, and reduces 80\% communication cost, with an up to 10x speedup to state-of-the-art approximated models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。