In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks with only a small set of examples at inference time, thereby avoiding task-specific fine-tuning. However, in-context examples may contain privacy-sensitive information that should not be revealed through model outputs. Existing differential privacy (DP) approaches to ICL are either computationally expensive or rely on heuristics with limited effectiveness, including context oversampling, synthetic data generation, or unnecessary thresholding. We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized. We evaluate our method across five datasets in text classification, math, and vision-language. We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.
Adaptive Expert Models for Personalization in Federated Learning
Ericsson AB(爱立信公司);KTH Royal Institute of Technology(KTH皇家理工学院);RISE Research Institutes of Sweden(瑞典RISE研究院);AI Sweden(瑞典人工智能研究院);Chalmers University of Technology(查尔姆斯理工大学)
DPOAD: Differentially Private Outsourcing of Anomaly Detection through Iterative Sensitivity Learning
Data61, CSIRO(澳大利亚联邦科学与工业研究组织数据61研究所);Cyber Security CRC(网络安全合作研究中心);Illinois Institute of Technology(伊利诺伊理工学院);Concordia University(康考迪亚大学);Hong Kong Polytechnic University(香港理工大学);Ericsson Security Research, Ericsson Canada(加拿大爱立信公司爱立信安全研究院);University of Connecticut(康涅狄格大学)
RAPTOR: Role-Aware Private Training for Mixture-of-Experts
Vector Institute(向量研究所);Knovel Engineering Lab(Knovel 工程实验室);KAIST(韩国科学技术院);MPI for Intelligent Systems, Tübingen(马克斯·普朗克智能系统研究所(蒂宾根));Salesforce AI Research(Salesforce AI 研究院);Carnegie Mellon University(卡内基梅隆大学);University of Toronto(多伦多大学);University of Oxford(牛津大学);Jinesis Lab, University of Toronto & Vector Institute(多伦多大学与向量研究所 Jinesis 实验室);Hanoi University of Science and Technology(河内理工大学);Vietnam National University, Hanoi(越南河内国家大学);University of Florida(佛罗里达大学);Stanford University(斯坦福大学);ELLIS Institute Tübingen(ELLIS 蒂宾根研究所);Northeastern University(东北大学)
Prediction with Expert Advice under Local Differential Privacy
Department of Computer Sciences University of Wisconsin — Madison(计算机科学系威斯康星大学麦迪逊分校);Department of Electrical and Computer Engineering University of Wisconsin — Madison(电气与计算机工程系威斯康星大学麦迪逊分校)
PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning
Beijing Univ. of Posts and Telecomm.(北京邮电大学);Beijing Jiaotong University(北京交通大学);Beijing Univ. of Aeronautics and Astronautics(北京航空航天大学);Tsinghua University(清华大学)