Mixture of Experts 模型的差分隐私训练
Differentially Private Training of Mixture of Experts Models
AI总结：
该论文首次探索在差分隐私约束下训练 Mixture of Experts (MoE) 模型，以应对大语言模型扩展带来的计算与隐私挑战，初步实验证明其性能可与非隐私模型相媲美。
摘要：
本立场论文探讨了在自然语言处理领域中，将差分隐私（DP）集成到 Mixture of Experts (MoE) 模型训练中的问题。随着大语言模型（LLMs）扩展至数十亿参数并利用海量数据集，它们展现出更强的语言能力和涌现能力。然而，这种增长引发了重大的计算和隐私担忧。我们的研究通过探索以计算效率著称的 MoE 模型的潜力，以及作为隐私保护标准的 DP 的应用来解决这些问题。我们提出了在 DP 约束下训练 MoE 模型的首次已知尝试，应对了由其架构和 DP 集成复杂性带来的独特挑战。我们的初步实验研究表明，MoE 模型可以在 DP 下进行有效训练，实现与非隐私模型相媲美的性能。这项初步研究旨在提供有价值的见解，并激发隐私保护 MoE 模型领域的进一步研究，为这一不断发展领域的未来进展奠定基础。
英文摘要：
This position paper investigates the integration of Differential Privacy (DP) in the training of Mixture of Experts (MoE) models within the field of natural language processing. As Large Language Models (LLMs) scale to billions of parameters, leveraging expansive datasets, they exhibit enhanced linguistic capabilities and emergent abilities. However, this growth raises significant computational and privacy concerns. Our study addresses these issues by exploring the potential of MoE models, known for their computational efficiency, and the application of DP, a standard for privacy preservation. We present the first known attempt to train MoE models under the constraints of DP, addressing the unique challenges posed by their architecture and the complexities of DP integration. Our initial experimental studies demonstrate that MoE models can be effectively trained with DP, achieving performance that is competitive with their non-private counterparts. This initial study aims to provide valuable insights and ignite further research in the domain of privacy-preserving MoE models, softly laying the groundwork for prospective developments in this evolving field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
