具有隐私保护的自然语言理解BERT
Natural Language Understanding with Privacy-Preserving BERT
摘要:
隐私保护仍然是数据采掘和自然语言理解(NLU)中的一个关键挑战。以往的研究表明,输入文本甚至文本嵌入都可能泄露隐私信息。这种关注激发了我们对预训练语言模型(LM)的有效隐私保护方法的研究。我们调查的隐私和实用程序的影响,应用dx隐私,局部差分隐私的变体,BERT微调在NLU应用程序。更重要的是,我们进一步提出了隐私自适应LM预训练方法,并表明我们的方法可以显着提高BERT的效用,同时保持相同的隐私保护水平。我们还量化了隐私保护的水平,并提供了隐私配置的指导。我们的实验和研究结果奠定了基础,为未来探索隐私保护NLU与预训练LM。
英文摘要:
Privacy preservation remains a key challenge in data mining and Natural Language Understanding (NLU). Previous research shows that the input text or even text embeddings can leak private information. This concern motivates our research on effective privacy preservation approaches for pretrained Language Models (LMs). We investigate the privacy and utility implications of applying dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU applications. More importantly, we further propose privacy-adaptive LM pretraining methods and show that our approach can boost the utility of BERT dramatically while retaining the same level of privacy protection. We also quantify the level of privacy preservation and provide guidance on privacy configuration. Our experiments and findings lay the groundwork for future explorations of privacy-preserving NLU with pretrained LMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。