PPC-GPT：通过剪枝与思维链蒸馏实现大型语言模型的联邦任务特定压缩
PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
AI总结：
PPC-GPT提出一种联邦框架，结合差分隐私、合成数据生成、剪枝与思维链蒸馏，将LLM压缩为任务特定SLM，在保护隐私的同时保持竞争性能。
摘要：
将大型语言模型（LLMs）压缩为任务特定的小型语言模型（SLMs）面临两大挑战：保护领域特定知识的隐私以及管理有限的资源。为应对这些挑战，我们提出PPC-GPT，这是一个新颖的统一框架，在联邦设置中系统地同时解决隐私保护和模型压缩问题。PPC-GPT基于服务器-客户端联邦架构运行，客户端将经过差分隐私（DP）扰动的任务特定数据发送到服务器的LLM。随后，LLM生成合成数据及其对应的推理依据。这些合成数据随后被用于LLM剪枝和再训练过程。我们框架的关键创新在于其整体性地整合了隐私保护机制、合成数据生成和任务特定压缩技术，通过组件间的相互作用创造了独特的优势。我们在多种文本生成任务上的实验表明，PPC-GPT成功实现了双重目标：在保持与全尺寸LLM相当的竞争性性能的同时，通过其联邦架构确保稳健的隐私保护。我们的代码已贡献给FATE开源项目，现已在\ extit{https://github.com/FederatedAI/FATE-LLM/tree/main/python/fate_llm/algo/ppc-gpt}公开访问。
英文摘要：
Compressing Large Language Models (LLMs) into task-specific Small Language Models (SLMs) encounters two significant challenges: safeguarding domain-specific knowledge privacy and managing limited resources. To tackle these challenges, we propose PPC-GPT, a novel unified framework that systematically addresses both privacy preservation and model compression in federated settings. PPC-GPT works on a server-client federated architecture, where the client sends differentially private (DP) perturbed task-specific data to the server's LLM. The LLM then generates synthetic data along with their corresponding rationales. This synthetic data is subsequently used for both LLM pruning and retraining processes. Our framework's key innovation lies in its holistic integration of privacy-preserving mechanisms, synthetic data generation, and task-specific compression techniques, creating unique benefits through component interaction. Our experiments across diverse text generation tasks demonstrate that PPC-GPT successfully achieves dual objectives: maintaining competitive performance comparable to full-sized LLMs while ensuring robust privacy protection through its federated architecture. Our code has been contributed to the FATE open-source project and is now publicly accessible at \textit{https://github.com/FederatedAI/FATE-LLM/tree/main/python/fate_llm/algo/ppc-gpt}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
