arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.18517cs.CRcs.AI
收藏

RewardDS:通过奖励驱动的数据合成实现大语言模型的隐私保护微调

RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis

South China University of Technology(华南理工大学)

Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng

AI总结:

针对大语言模型微调中的隐私数据泄露风险与现有差分隐私合成数据噪声大的问题,提出含奖励引导过滤、自优化精炼模块的RewardDS框架,用奖励信号引导合成数据生成与优化,经多领域实验验证有效。

摘要:

大语言模型(LLM)的成功吸引了众多用户上传数据,针对领域特定任务对模型进行微调。但在医疗、金融等敏感领域,这类操作常引发隐私担忧。一种颇具前景的解决方案是生成具备差分隐私(DP,Differential Privacy)保障的合成数据,以替代私有数据,然而这类合成数据包含大量被视为噪声的缺陷数据。现有方案通常依赖比较ROUGE-L分数或嵌入相似度的简单过滤方式,无法有效消除这类噪声。为解决该问题,我们提出RewardDS——一种新型隐私保护框架,该框架微调奖励代理模型,并利用奖励信号引导合成数据生成。RewardDS包含两个核心模块:奖励引导过滤与自优化精炼,可同时对合成数据进行过滤和精炼,有效缓解噪声问题。我们在医疗、金融、代码生成三个领域开展了大量实验,结果验证了该方法的有效性。

英文摘要:

The success of large language models (LLMs) has attracted many individuals to fine-tune them for domain-specific tasks by uploading their data. However, in sensitive areas like healthcare and finance, privacy concerns often arise. One promising solution is to generate synthetic data with Differential Privacy (DP) guarantees to replace private data. However, these synthetic data contain significant flawed data, which are considered as noise. Existing solutions typically rely on naive filtering by comparing ROUGE-L scores or embedding similarities, which are ineffective in addressing the noise. To address this issue, we propose \textit{RewardDS}, a novel privacy-preserving framework that fine-tunes a reward proxy model and uses reward signals to guide the synthetic data generation. Our \textit{RewardDS} introduces two key modules, Reward Guided Filtering and Self-Optimizing Refinement, to both filter and refine the synthetic data, effectively mitigating the noise. Extensive experiments across medical, financial, and code generation domains demonstrate the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑