SentinelLMs：加密输入适配与语言模型微调，实现私有和安全推理
SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inference
AI总结：
本文提出SentinelLMs方法，通过对分词器和词元嵌入进行不可逆变换并微调，使语言模型能直接处理加密输入，在保护隐私的同时保持与原始模型相当的性能。
摘要：
本文解决了深度神经语言模型相关的隐私和安全问题，这些模型是现代各类基于AI的应用中的关键组成部分。这些模型通常在预训练后针对特定任务进行微调，并通过互联网在服务器上部署以供访问。然而，这引入了两个基本风险：（a）用户输入通过网络传输到服务器会产生拦截漏洞，（b）部署此类模型的组织在受限上下文中存储用户数据，从而引发隐私问题。为解决这一问题，我们提出了一种新方法，用于在基于口令密钥加密的用户特定文本上适配和微调基于Transformer的语言模型。原始预训练语言模型首先经历快速适配（无需任何进一步预训练），对分词器和词元嵌入应用一系列不可逆变换。这使得模型能够对加密输入执行推理，同时防止从模型参数和中间输出中逆向工程出文本。适配后，模型在现有训练数据集的加密版本上进行微调。实验评估采用知名模型（如BERT、RoBERTa）的适配版本，在文本分类和序列标注的既定基准英语和多语言数据集上进行，结果表明加密模型与其原始对应模型实现了性能持平。这有助于协同保障性能、隐私和安全性。
英文摘要：
This paper addresses the privacy and security concerns associated with deep neural language models, which serve as crucial components in various modern AI-based applications. These models are often used after being pre-trained and fine-tuned for specific tasks, with deployment on servers accessed through the internet. However, this introduces two fundamental risks: (a) the transmission of user inputs to the server via the network gives rise to interception vulnerabilities, and (b) privacy concerns emerge as organizations that deploy such models store user data with restricted context. To address this, we propose a novel method to adapt and fine-tune transformer-based language models on passkey-encrypted user-specific text. The original pre-trained language model first undergoes a quick adaptation (without any further pre-training) with a series of irreversible transformations applied to the tokenizer and token embeddings. This enables the model to perform inference on encrypted inputs while preventing reverse engineering of text from model parameters and intermediate outputs. After adaptation, models are fine-tuned on encrypted versions of existing training datasets. Experimental evaluation employing adapted versions of renowned models (e.g., BERT, RoBERTa) across established benchmark English and multilingual datasets for text classification and sequence labeling shows that encrypted models achieve performance parity with their original counterparts. This serves to safeguard performance, privacy, and security cohesively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
