在加密合成数据上进行持续预训练以实现隐私保护的大语言模型
Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs
AI总结：
本文提出了一种基于实体的加密合成数据预训练框架，通过加密保护PII，实现隐私保护的大语言模型预训练，同时保持模型的指令遵循能力。
摘要：
在敏感数据上保护隐私的同时，对小规模、领域特定语料库进行大语言模型预训练是一项重大挑战。在本工作中，我们通过提出一个基于实体的框架，探索隐私保护的持续预训练方法，该框架合成加密训练数据以保护个人身份信息（PII）。我们的方法构建了一个加权实体图来指导数据合成，并对PII实体应用确定性加密，使LLM通过持续预训练编码新知识，同时通过解密密钥授权访问敏感数据。我们在有限规模的数据集上的结果表明，我们的预训练模型优于基础模型，并确保PII安全，同时与未加密合成数据训练的模型相比表现出适度的性能差距。我们进一步显示，增加实体数量并利用基于图的合成可以提高模型性能，且加密模型在长检索上下文中保留了遵循指令的能力。我们讨论了确定性加密的安全影响和限制，并将本工作定位为加密数据预训练设计空间的初步研究，用于隐私保护的LLM。我们的代码可在https://github.com/DataArcTech/SoE上获得。
英文摘要：
Preserving privacy in sensitive data while pretraining large language models on small, domain-specific corpora presents a significant challenge. In this work, we take an exploratory step toward privacy-preserving continual pretraining by proposing an entity-based framework that synthesizes encrypted training data to protect personally identifiable information (PII). Our approach constructs a weighted entity graph to guide data synthesis and applies deterministic encryption to PII entities, enabling LLMs to encode new knowledge through continual pretraining while granting authorized access to sensitive data through decryption keys. Our results on limited-scale datasets demonstrate that our pretrained models outperform base models and ensure PII security, while exhibiting a modest performance gap compared to models trained on unencrypted synthetic data. We further show that increasing the number of entities and leveraging graph-based synthesis improves model performance, and that encrypted models retain instruction-following capabilities with long retrieved contexts. We discuss the security implications and limitations of deterministic encryption, positioning this work as an initial investigation into the design space of encrypted data pretraining for privacy-preserving LLMs. Our code is available at https://github.com/DataArcTech/SoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
