测试时强化学习中的放大效应：安全性和推理漏洞
Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities
AI总结：
本文研究了测试时训练方法的安全性漏洞，发现测试时强化学习中有害提示注入会放大模型行为，导致推理能力下降。
摘要：
测试时训练（TTT）最近作为一种提升大语言模型（LLMs）推理能力的有前景方法出现，其中模型直接从测试数据学习而无需标签。然而，这种依赖测试数据也使TTT方法容易受到有害提示注入的攻击。本文研究了TTT方法的安全性漏洞，探讨了代表性自我一致性基于测试时学习方法：测试时强化学习（TTRL），一种近期TTT方法，通过使用多数投票作为奖励信号来改进LLM推理。我们发现，有害提示注入期间的TTRL会放大模型的现有行为，即当基础模型相对安全时为安全放大，当其易受注入数据攻击时为有害放大。在两种情况下，推理能力都会下降，我们称之为推理税。我们还展示，如TTRL这样的TTT方法可以被使用特别设计的“HarmInject”提示进行对抗性利用，迫使模型同时回答越狱和推理查询，从而产生更强的有害放大。总体而言，我们的结果表明，通过促进自我一致性来增强LLM推理的TTT方法可能导致放大行为和推理退化，凸显了需要更安全的TTT方法的必要性。
英文摘要：
Test-time training (TTT) has recently emerged as a promising method to improve the reasoning abilities of large language models (LLMs), in which the model directly learns from test data without access to labels. However, this reliance on test data also makes TTT methods vulnerable to harmful prompt injections. In this paper, we investigate safety vulnerabilities of TTT methods, where we study a representative self-consistency-based test-time learning method: test-time reinforcement learning (TTRL), a recent TTT method that improves LLM reasoning by rewarding self-consistency using majority vote as a reward signal. We show that harmful prompt injection during TTRL amplifies the model's existing behaviors, i.e., safety amplification when the base model is relatively safe, and harmfulness amplification when it is vulnerable to the injected data. In both cases, there is a decline in reasoning ability, which we refer to as the reasoning tax. We also show that TTT methods such as TTRL can be exploited adversarially using specially designed "HarmInject" prompts to force the model to answer jailbreak and reasoning queries together, resulting in stronger harmfulness amplification. Overall, our results highlight that TTT methods that enhance LLM reasoning by promoting self-consistency can lead to amplification behaviors and reasoning degradation, highlighting the need for safer TTT methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
