用于对齐大型语言模型的隐私保护指令
Privacy-Preserving Instructions for Aligning Large Language Models
AI总结：
针对用户指令人工标注带来的隐私风险，本文提出用私有微调生成器产生具有差分隐私保证的合成指令，并通过分布匹配过滤保持效用，实验表明其可媲美真实指令且优于 Vicuna。
摘要：
大型语言模型（LLM）应用的服务提供商会从真实环境中收集用户指令，并将其用于进一步使 LLM 与用户意图对齐。这些指令可能包含敏感信息，并且在此过程中由人工工作者进行标注。这带来了典型的私有优化尚未解决的新型隐私风险。为此，我们提出在数据标注和模型微调中使用合成指令替代真实指令。通过使用经过私有微调的生成器生成这些合成指令，可保证形式化的差分隐私。实现预期效用的关键在于我们提出的新型过滤算法，该算法使合成指令的分布与真实指令的分布相匹配。在监督微调和基于人类反馈的强化学习中，我们的大量实验通过展示与真实指令相当的结果，证明了最终合成指令集的高效用。在监督微调中，使用私有合成指令训练的模型优于 Vicuna 等领先的开源模型。
英文摘要：
Service providers of large language model (LLM) applications collect user instructions in the wild and use them in further aligning LLMs with users' intentions. These instructions, which potentially contain sensitive information, are annotated by human workers in the process. This poses a new privacy risk not addressed by the typical private optimization. To this end, we propose using synthetic instructions to replace real instructions in data annotation and model fine-tuning. Formal differential privacy is guaranteed by generating those synthetic instructions using privately fine-tuned generators. Crucial in achieving the desired utility is our novel filtering algorithm that matches the distribution of the synthetic instructions to that of the real ones. In both supervised fine-tuning and reinforcement learning from human feedback, our extensive experiments demonstrate the high utility of the final set of synthetic instructions by showing comparable results to real instructions. In supervised fine-tuning, models trained with private synthetic instructions outperform leading open-source models such as Vicuna.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
