arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.16451cs.LGcs.AIcs.CL
收藏

通过影响函数纠正大型语言模型行为

Correcting Large Language Model Behavior via Influence Function

Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Pengcheng Laboratory(鹏城实验室);National University of Defense Technology(国防科技大学);The Chinese University of Hong Kong(香港中文大学);King’s College London(伦敦国王学院)

Han Zhang, Zhuo Zhang, Yi Zhang, Yuanzhao Zhai, Hanyang Peng, Yu Lei, Yue Yu, Hui Wang, Bin Liang, Lin Gui, Ruifeng Xu

AI总结:

本文提出无需人类参与的LANCET方法,通过影响函数识别不良数据并利用IBO技术调整模型行为,高效纠正LLMs的不当行为,优于依赖人工偏好的方法并提升可解释性。

摘要:

近期AI对齐技术的进步显著提升了大型语言模型(LLMs)与静态人类偏好的对齐度。然而,人类偏好的动态特性可能使部分先前的训练数据过时甚至错误,最终导致LLMs偏离当代人类偏好和社会规范。现有方法无论涉及策划新数据以持续对齐,还是手动纠正过时数据以重新对齐,都需要昂贵的人力资源。为应对此挑战,我们提出一种无需人类参与的新方法:结合影响函数召回与后训练的大型语言模型行为纠正(LANCET)。LANCET包含两个阶段:(1)利用影响函数识别对不良模型输出产生显著影响的训练数据;(2)应用影响函数驱动的Bregman优化(IBO)技术,根据这些影响分布调整模型行为。实验表明,LANCET能高效有效地纠正LLMs的不当行为。此外,LANCET的表现优于依赖收集人类偏好的方法,并提升了LLMs中学习人类偏好的可解释性。

英文摘要:

Recent advancements in AI alignment techniques have significantly improved the alignment of large language models (LLMs) with static human preferences. However, the dynamic nature of human preferences can render some prior training data outdated or even erroneous, ultimately causing LLMs to deviate from contemporary human preferences and societal norms. Existing methodologies, whether they involve the curation of new data for continual alignment or the manual correction of outdated data for re-alignment, demand costly human resources. To address this challenge, we propose a novel approach, Large Language Model Behavior Correction with Influence Function Recall and Post-Training (LANCET), which requires no human involvement. LANCET consists of two phases: (1) using influence functions to identify the training data that significantly impact undesirable model outputs, and (2) applying an Influence function-driven Bregman Optimization (IBO) technique to adjust the model's behavior based on these influence distributions. Our experiments demonstrate that LANCET effectively and efficiently correct inappropriate behaviors of LLMs. Furthermore, LANCET can outperform methods that rely on collecting human preferences, and it enhances the interpretability of learning human preferences within LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑