DP-RFT: 通过差分隐私强化微调学习生成合成文本
DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning
AI总结:
DP-RFT通过差分隐私强化微调方法,在不直接接触隐私数据的情况下生成高质量合成文本。
摘要:
差分隐私(DP)合成数据生成在开发基于隐私数据的大型语言模型(LLMs)中发挥着关键作用,其中数据拥有者无法提供对单个示例的直接访问。生成DP合成数据通常涉及一个困难的权衡。一方面,DP微调方法通过正式的隐私保证训练LLM作为合成数据生成器,但仍然需要私人示例的原始内容进行模型训练。然而,避免直接接触隐私数据的方法受限于未经微调的现成模型,其输出通常缺乏领域保真度。能否训练LLM生成高质量的合成文本而无需对单个隐私示例进行直接访问?在本文中,我们引入了差分隐私强化微调(DP-RFT),一种用于合成数据生成的在线强化学习算法,使用DP保护的最近邻投票作为LLM生成的合成样本的奖励信号。LLM通过近端策略优化(PPO)迭代学习生成合成数据,以最大化预期的DP投票。我们评估了DP-RFT在长文本和领域特定合成数据生成中的表现,如新闻文章、会议记录和医学文章摘要。我们的实验表明,DP-RFT在保真度和下游效用方面缩小了隐私演化与DP微调方法之间的差距,同时尊重隐私数据边界。
英文摘要:
Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。