超越语义操控:奖励模型中的标记空间攻击
Fooling Reward Models with Gibberish: Adversarial Examples Beyond the Semantic Regime
AI总结:
本文提出Token Mapping Perturbation Attack(TOMPA)框架,通过在标记空间中进行对抗优化,发现非语言标记模式以提升奖励模型性能,揭示了当前RLHF流程的关键漏洞。
摘要:
奖励模型(RMs)广泛用于强化学习从人类反馈(RLHF)中的优化目标,但仍然易受奖励黑客攻击。现有攻击主要在语义空间中操作,构建可读的对抗性输出以利用RM偏见。本文引入了一种不同的范式:标记映射扰动攻击(TOMPA),一个在标记空间中进行对抗优化的框架。通过绕过策略与奖励模型之间的标准解码-重标记接口,TOMPA使攻击策略能够在原始标记序列上优化,而非连贯的自然语言。仅使用黑盒标量反馈,TOMPA自动发现非语言标记模式,这些模式在多个最先进的RMs上产生极高的奖励。具体而言,当针对Skywork-Reward-V2-Llama-3.1-8B时,TOMPA几乎将GPT-5参考答案的奖励翻倍,并在98.0%的提示上优于它们。尽管这些高分,生成的输出退化为无意义文本,揭示了RMs可以系统性地被利用超出语义领域,并暴露了当前RLHF流程中的关键漏洞。
英文摘要:
Reward models (RMs) are widely used as optimization targets in reinforcement learning from human feedback (RLHF). We show that leading RMs can assign higher scores to gibberish than to meaningful reference answers. To systematically discover such inputs, we introduce Token Mapping Perturbation Attack (TOMPA), a framework that uses a perturbed token mapping to extend an attack LLM's search beyond natural language. The mapping changes the linguistic meaning of the attack LLM's natural language responses, producing non-linguistic text for the target RM to score. We then use these scalar scores as rewards to train the attack LLM through reinforcement learning to discover non-linguistic adversarial inputs. Experiments against Llama and Qwen RMs demonstrate such attacks using two instantiations: identity token mapping across different tokenizers and random permutation with a shared tokenizer. Against Skywork-Reward-V2-Llama-3.1-8B, ranked first on RewardBench 2, the two mappings achieve mean rewards of +33.64 and +43.78, respectively, compared with +17.48 for GPT-5 reference answers. Their attack success rates are 97.6% and 100%, measured as the percentages of individual responses scoring above their corresponding GPT-5 references. These results show that strong benchmark performance does not ensure robustness to non-linguistic adversarial inputs, exposing a fundamental vulnerability in using RMs as optimization targets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。