arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.22971cs.CL
收藏

基于扩散模型的私有合成文本生成

Private Synthetic Text Generation with Diffusion Models

Technical University of Darmstadt(达姆施塔特工业大学);University Alliance Ruhr(鲁尔大学联盟);Ruhr University Bochum(波鸿鲁尔大学)

Sebastian Ochs, Ivan Habernal

AI总结:

本文针对差分隐私训练下扩散模型的合成文本生成能力开展实验,复现评估过往LLM私有文本生成研究并发现其隐私保障漏洞,证实隐私场景下开源LLM表现优于扩散模型,同时公开了相关实验资源。

摘要:

扩散模型生成合成文本的能力如何?近期研究显示其具备优势,性能已达自回归大语言模型(LLM)水平。但如果在差分隐私约束下训练,它们是否仍擅长生成合成数据?目前相关证据尚且缺失,不过私有图像生成领域的发展前景十分可观。本文通过大量实验探究这一开放性问题,同时对以往基于LLM的私有合成文本生成研究进行严格评估与复现,发现其中部分未满足的假设可能导致差分隐私保障被破坏。研究结果与以往非私有场景下的结论部分相悖,表明在隐私约束场景下,完全开源的LLM表现优于扩散模型。我们已公开全部源代码、数据集与实验设置,以推动未来相关研究。

英文摘要:

How capable are diffusion models of generating synthetics texts? Recent research shows their strengths, with performance reaching that of auto-regressive LLMs. But are they also good in generating synthetic data if the training was under differential privacy? Here the evidence is missing, yet the promises from private image generation look strong. In this paper we address this open question by extensive experiments. At the same time, we critically assess (and reimplement) previous works on synthetic private text generation with LLMs and reveal some unmet assumptions that might have led to violating the differential privacy guarantees. Our results partly contradict previous non-private findings and show that fully open-source LLMs outperform diffusion models in the privacy regime. Our complete source codes, datasets, and experimental setup is publicly available to foster future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑