arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.03371cs.CRcs.AI
收藏

睡眠细胞:将潜在恶意时间后门注入工具使用的大语言模型

Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs

Bhanu Pallakonda, Mikkel Hindsbo, Sina Ehsani, Prag Mishra

AI总结:

本研究提出SFT-then-GRPO方法,通过多阶段PEFT框架在工具使用LLM中植入隐蔽的恶意后门,揭示了强化学习在掩盖灾难性漏洞中的潜在风险。

摘要:

开放式大型语言模型(LLM)的普及使代理AI民主化,但微调权重经常被共享和采用,且在排行榜性能之外受到有限审查。这导致第三方模型在缺乏强行为保证的情况下被纳入其中。在本工作中,我们展示了一种新的向量用于隐蔽后门注入:通过多阶段参数高效微调(PEFT)框架在使用工具的代理中植入潜在恶意行为。我们的方法,SFT-then-GRPO,将能力注入与行为对齐解耦。首先,我们使用LoRA进行SFT来植入“睡眠代理”能力。其次,我们应用具有专门奖励函数的组相对策略优化(GRPO)来强制执行欺骗性策略。这强化了两种行为:(1)触发特定性,严格限制执行到目标条件(例如,2026年),以及(2)操作隐蔽性,其中模型在破坏性行为后立即生成良性文本响应。我们实证表明,这些被污染的模型在良性任务上保持最先进的性能,激励其采用。我们的发现突显了对齐中的关键失败模式,其中强化学习被用来掩盖,而不是消除,灾难性漏洞。我们最后讨论了潜在的识别策略,重点关注标准基准的差异和随机探测以揭示这些潜在威胁。

英文摘要:

The proliferation of open-weight Large Language Models (LLMs) has democratized agentic AI, yet fine-tuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance. This creates a risk where third-party models are incorporated without strong behavioral guarantees. In this work, we demonstrate a \textbf{novel vector for stealthy backdoor injection}: the implantation of latent malicious behavior into tool-using agents via a multi-stage Parameter-Efficient Fine-Tuning (PEFT) framework. Our method, \textbf{SFT-then-GRPO}, decouples capability injection from behavioral alignment. First, we use SFT with LoRA to implant a "sleeper agent" capability. Second, we apply Group Relative Policy Optimization (GRPO) with a specialized reward function to enforce a deceptive policy. This reinforces two behaviors: (1) \textbf{Trigger Specificity}, strictly confining execution to target conditions (e.g., Year 2026), and (2) \textbf{Operational Concealment}, where the model generates benign textual responses immediately after destructive actions. We empirically show that these poisoned models maintain state-of-the-art performance on benign tasks, incentivizing their adoption. Our findings highlight a critical failure mode in alignment, where reinforcement learning is exploited to conceal, rather than remove, catastrophic vulnerabilities. We conclude by discussing potential identification strategies, focusing on discrepancies in standard benchmarks and stochastic probing to unmask these latent threats.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑