arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.03039cs.LGcs.AI
收藏

通过对抗性RLHF平台导致的LLM错位

LLM Misalignment via Adversarial RLHF Platforms

Erfan Entezami, Ali Naseh

AI总结:

本文提出一种针对公开RLHF平台的对抗性攻击,通过选择性操纵偏好数据集损坏奖励模型,导致语言模型在目标领域内发生错位并产生不良行为,揭示了RLHF平台的安全漏洞。

摘要:

强化学习在将语言模型与人类偏好对齐方面表现出了卓越的性能,引发了对开发RLHF平台的关注。这些平台使用户能够微调模型,而无需具备开发复杂机器学习算法的专业知识。尽管这些平台提供了奖励建模和RLHF微调等有用功能,但其安全性和可靠性在很大程度上仍未被探索。鉴于RLHF和开源RLHF框架的日益普及,我们调查了这些系统的可信度及其对LLM行为的潜在影响。在本文中,我们提出了一种针对公开可用RLHF工具的攻击。在我们提出的攻击中,对抗性RLHF平台通过选择性地操纵偏好数据集中的数据样本来破坏LLM对齐过程。在这种情况下,当用户的任务与攻击者的目标一致时,平台会操纵偏好数据集中包含与攻击者目标相关样本的子集。这种操纵会导致奖励模型损坏,最终导致语言模型错位。我们的结果表明,此类攻击能够有效地引导LLM在目标领域内表现出不良行为。我们的工作强调了探索RLHF平台漏洞及其在RLHF微调过程中导致LLM错位的潜在能力的迫切需求。

英文摘要:

Reinforcement learning has shown remarkable performance in aligning language models with human preferences, leading to the rise of attention towards developing RLHF platforms. These platforms enable users to fine-tune models without requiring any expertise in developing complex machine learning algorithms. While these platforms offer useful features such as reward modeling and RLHF fine-tuning, their security and reliability remain largely unexplored. Given the growing adoption of RLHF and open-source RLHF frameworks, we investigate the trustworthiness of these systems and their potential impact on behavior of LLMs. In this paper, we present an attack targeting publicly available RLHF tools. In our proposed attack, an adversarial RLHF platform corrupts the LLM alignment process by selectively manipulating data samples in the preference dataset. In this scenario, when a user's task aligns with the attacker's objective, the platform manipulates a subset of the preference dataset that contains samples related to the attacker's target. This manipulation results in a corrupted reward model, which ultimately leads to the misalignment of the language model. Our results demonstrate that such an attack can effectively steer LLMs toward undesirable behaviors within the targeted domains. Our work highlights the critical need to explore the vulnerabilities of RLHF platforms and their potential to cause misalignment in LLMs during the RLHF fine-tuning process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑