arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.01658cs.CL
收藏

从唯唯诺诺者到直言不讳者:用精准微调解决大语言模型中的谄媚问题

From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning

School of Software Technology, Zhejiang University(浙江大学软件学院);Alibaba Cloud(阿里云);Hong Kong Baptist University(香港浸会大学);Fullong Inc.(福隆公司);University of Science and Technology of China(中国科学技术大学);Zhejiang University of Technology(浙江工业大学)

Wei Chen, Zhen Huang, Liang Xie, Binbin Lin, Houqiang Li, Le Lu, Xinmei Tian, Deng Cai, Yonggang Zhang, Wenxiao Wang, Xu Shen, Jieping Ye

AI总结:

本文提出监督精准微调(SPT)方法,通过仅微调不足5%的关键模块来缓解大语言模型的谄媚问题,在保持通用能力的同时效果优于传统监督微调。

摘要:

大语言模型(LLMs)倾向于优先遵循用户提示,而不是提供真实回答,从而导致谄媚问题。当受到用户质疑时,LLMs 倾向于承认错误并提供不准确的回答,即使它们最初给出了正确答案。最近的工作提出采用监督微调(SFT)来缓解谄媚问题,但这通常会导致 LLMs 通用能力的退化。为了解决这一挑战,我们提出了一种新颖的监督精准微调(SPT),其中针对给定目标对感兴趣区域的模块进行微调。具体来说,SPT 首先揭示并验证一小部分(<5%)的基础模块,这些模块显著影响 LLMs 的特定行为,即谄媚。随后,SPT 仅微调这些已识别的模块,同时冻结其余模块。为了验证所提出的 SPT 的有效性,我们进行了全面的实验,证明 SPT 显著缓解了 LLMs 的谄媚问题(甚至优于 SFT)。此外,SPT 对 LLMs 的通用能力产生的副作用有限甚至没有副作用。我们的结果揭示了如何精确、有效且高效地解释和提升 LLMs 的目标能力。代码和数据可在 https://github.com/yellowtownhz/sycophancy-interpretability 获取。

英文摘要:

Large Language Models (LLMs) tend to prioritize adherence to user prompts over providing veracious responses, leading to the sycophancy issue. When challenged by users, LLMs tend to admit mistakes and provide inaccurate responses even if they initially provided the correct answer. Recent works propose to employ supervised fine-tuning (SFT) to mitigate the sycophancy issue, while it typically leads to the degeneration of LLMs' general capability. To address the challenge, we propose a novel supervised pinpoint tuning (SPT), where the region-of-interest modules are tuned for a given objective. Specifically, SPT first reveals and verifies a small percentage (<5%) of the basic modules, which significantly affect a particular behavior of LLMs. i.e., sycophancy. Subsequently, SPT merely fine-tunes these identified modules while freezing the rest. To verify the effectiveness of the proposed SPT, we conduct comprehensive experiments, demonstrating that SPT significantly mitigates the sycophancy issue of LLMs (even better than SFT). Moreover, SPT introduces limited or even no side effects on the general capability of LLMs. Our results shed light on how to precisely, effectively, and efficiently explain and improve the targeted ability of LLMs. Code and data are available at https://github.com/yellowtownhz/sycophancy-interpretability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑