模型手术：通过简单参数编辑调控大语言模型的行为
Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
AI总结：
针对LLM部署需理想行为但现有微调方法成本高、易损基础能力的问题，提出“模型手术”直接编辑小参数子集，以推理级成本实现无毒化等行为调控，实验验证了其在降低毒性同时保持通用能力的效果。
摘要：
大语言模型（LLMs）已展现出作为通用助手的巨大潜力，具备强大的任务理解与问题解决能力。要将LLMs部署为AI助手，模型展现出理想行为特征（如无毒化、抗越狱攻击能力）至关重要。当前用于无毒化或防止越狱的方法通常涉及监督微调（SFT）或基于人类反馈的强化学习（RLHF），这些方法需通过梯度下降微调数十亿参数，计算成本高昂。此外，经SFT和RLHF修改的模型可能偏离预训练模型，或导致LLM基础能力下降。本文观察到，仅用推理级计算资源直接编辑小部分参数，即可有效调控LLMs的特定行为（如无毒化、抗越狱）。实验显示，在无毒化任务中，该方法在RealToxicityPrompts数据集上降低毒性达90.0%，在ToxiGen上达49.2%，同时保持LLM在常识、问答、数学等领域的通用能力。
英文摘要：
Large Language Models (LLMs) have demonstrated great potential as generalist assistants, showcasing powerful task understanding and problem-solving capabilities. To deploy LLMs as AI assistants, it is crucial that these models exhibit desirable behavioral traits, such as non-toxicity and resilience against jailbreak attempts. Current approaches for detoxification or preventing jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), which requires finetuning billions of parameters through gradient descent with substantial computational cost. Furthermore, models modified through SFT and RLHF may deviate from the pretrained models, potentially leading to a degradation in foundational LLM capabilities. In this paper, we observe that surprisingly, directly editing a small subset of parameters can effectively modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreaking, with only inference-level computational resources. Experiments demonstrate that in the detoxification task, our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen, while maintaining the LLM's general capabilities in areas such as common sense, question answering, and mathematics
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
