Large language models (LLMs) offer personalized responses based on user interactions, but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a cryptographic protocol supporting arithmetic computations in encrypted states and provides a potential solution for privacy-preserving machine learning (PPML). However, the computational intensity of transformers poses challenges for applying HE to LLMs. In this work, we propose a modified HE-friendly transformer architecture with an emphasis on inference following personalized (private) fine-tuning. Utilizing LoRA fine-tuning and Gaussian kernels, we achieve significant computational speedups -- 6.94x for fine-tuning and 2.3x for inference -- while maintaining performance comparable to plaintext models. Our findings provide a viable proof of concept for offering privacy-preserving LLM services in areas where data protection is crucial. Our code is available on GitHub.
Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference
Chongqing Key Laboratory of Secure Computing for Biology（重庆生物安全计算重点实验室）；Chongqing Institute of Green and Intelligent Technology（重庆绿色智能技术研究所）；Chinese Academy of Sciences（中国科学院）；Department of Industrial and Systems Engineering（工业与系统工程系）
Recently Homomorphic Encryption (HE) is used to implement Privacy-Preserving Neural Networks (PPNNs) that perform inferences directly on encrypted data without decryption. Prior PPNNs adopt mobile network architectures such as SqueezeNet for smaller computing overhead, but we find naïvely using mobile network architectures for a PPNN does not necessarily achieve shorter inference latency. Despite having less parameters, a mobile network architecture typically introduces more layers and increases the HE multiplicative depth of a PPNN, thereby prolonging its inference latency. In this paper, we …
The University of Manchester（曼彻斯特大学）；KU Leuven（鲁汶大学）；The University of British Columbia（不列颠哥伦比亚大学）
In this paper, we propose a decentralized, privacy-friendly energy trading platform (PFET) based on game theoretical approach - specifically Stackelberg competition. Unlike existing trading schemes, PFET provides a competitive market in which prices and demands are determined based on competition, and computations are performed in a decentralized manner which does not rely on trusted third parties. It uses homomorphic encryption cryptosystem to encrypt sensitive information of buyers and sellers such as sellers$'$ prices and buyers$'$ demands. Buyers calculate total demand on particular seller…
Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs
The PII shown in the figure is synthetic and not real（合成的PII）；International Digital Economy Academy（国际数字经济学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；DataArc Tech Ltd（DataArc科技有限公司）
Privacy-Preserving Product-Quantized Approximate Nearest Neighbor Search Framework for Large-scale Datasets via A Hybrid of Fully Homomorphic Encryption and Trusted Execution Environment
CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges
Shanghai AI Laboratory（上海人工智能实验室）；Wuhan University（武汉大学）；Renmin University of China（中国人民大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）
RAMSES: Secure high-performance computing for sensitive data
IT Center Cologne (ITCC) University of Cologne（科隆大学计算中心）；NEC Deutschland GmbH（日本电气德国有限公司）；Department of Translational Genomics University of Cologne（科隆大学转化基因组学系）
