在RLVR中存在后门：从可验证奖励中对LLM的后门攻击
Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward
AI总结：
本文提出了一种在RLVR框架中通过注入污染数据植入后门的新方法，展示了该攻击在不同模型规模上高效且具有泛化能力，导致安全性能下降73%。
摘要：
可验证奖励强化学习（RLVR）是一种新兴范式，显著提升了大语言模型（LLM）在复杂逻辑任务上的推理能力，如数学和编程。然而，我们首次发现RLVR框架中存在潜在的后门攻击漏洞。该攻击可通过向训练集注入少量污染数据，而无需修改奖励验证器，植入后门。具体而言，我们提出了一种新的触发机制，称为ACB。攻击利用RLVR训练循环，对有害响应给予大量正奖励，对拒绝给予负奖励。这种不对称的奖励信号迫使模型在训练过程中逐步增加生成有害响应的概率。我们的发现表明，RLVR后门攻击具有高效率和强泛化能力。使用不到2%的污染数据，在训练集上植入后门，不会影响良性任务的性能。在多个后门基准测试中的评估表明，激活触发器使安全性能平均下降73%。此外，该攻击有效泛化到广泛的后门方法和不安全行为。代码可在https://github.com/yuki-younai/Backdoor_in_RLVR上获得。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) is an emerging paradigm that significantly boosts a Large Language Model's (LLM's) reasoning abilities on complex logical tasks, such as mathematics and programming. However, we identify, for the first time, a latent vulnerability to backdoor attacks within the RLVR framework. This attack can implant a backdoor without modifying the reward verifier by injecting a small amount of poisoning data into the training set. Specifically, we propose a novel trigger mechanism designated as the \ourapproach (ACB). The attack exploits the RLVR training loop by assigning substantial positive rewards for harmful responses and negative rewards for refusals. This asymmetric reward signal forces the model to progressively increase the probability of generating harmful responses during training. Our findings demonstrate that the RLVR backdoor attack is characterized by both high efficiency and strong generalization capabilities. Utilizing less than 2\% poisoned data in train set, the backdoor can be successfully implanted across various model scales without degrading performance on benign tasks. Evaluations across multiple jailbreak benchmarks indicate that activating the trigger degrades safety performance by an average of 73\%. Furthermore, the attack generalizes effectively to a wide range of jailbreak methods and unsafe behaviors. Code is available at https://github.com/yuki-younai/Backdoor_in_RLVR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
