arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18329cs.AI
收藏

FaithSteer-BENCH:一种面向部署的应力测试基准,用于推理时的操控

FaithSteer-BENCH: A Deployment-Aligned Stress-Testing Benchmark for Inference-Time Steering

University of Electronic Science and Technology of China(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学);South China University of Technology(华南理工大学)

Zikang Ding, Qiying Hu, Yi Zhang, Hongji Li, Junchi Yao, Hongbo Liu, Lijie Hu

AI总结:

本文提出FaithSteer-BENCH基准,通过可控性、效用保持和鲁棒性三个指标评估推理时操控方法,揭示现有方法在部署场景下的可靠性问题及机制层面的对齐问题。

摘要:

推理时的操控被广泛认为是一种轻量且无参数的机制,用于控制大语言模型(LLM)的行为。先前的工作通常认为简单的激活层干预可以可靠地引起目标行为变化。然而,这些结论通常是在相对较宽松的评估设置下得出的,忽略了部署约束、能力权衡和现实世界鲁棒性。因此,我们引入FaithSteer-BENCH,一种应力测试基准,通过三个门控指标评估操控方法在固定部署风格的操作点上的性能。在多个模型和代表性操控方法上,我们揭示了若干系统性失效模式,这些模式在标准评估下往往被掩盖,包括虚假可控性、可测量的认知税在无关能力上的消耗以及在轻微指令级扰动、角色提示、编码转换和数据稀缺下的显著脆弱性。门控基准结果表明,现有方法在面向部署的实用场景中并不一定提供可靠的可控性。此外,机制层面的诊断表明,许多操控方法诱导了提示条件对齐,而不是稳定的潜在方向偏移,进一步解释了其在压力下的脆弱性。FaithSteer-BENCH因此提供了一个统一的基准和更清晰的分析视角,用于未来方法设计、可靠性评估和面向部署的研究。

英文摘要:

Inference-time steering is widely regarded as a lightweight and parameter-free mechanism for controlling large language model (LLM) behavior, and prior work has often suggested that simple activation-level interventions can reliably induce targeted behavioral changes. However, such conclusions are typically drawn under relatively relaxed evaluation settings that overlook deployment constraints, capability trade-offs, and real-world robustness. We therefore introduce \textbf{FaithSteer-BENCH}, a stress-testing benchmark that evaluates steering methods at a fixed deployment-style operating point through three gate-wise criteria: controllability, utility preservation, and robustness. Across multiple models and representative steering approaches, we uncover several systematic failure modes that are largely obscured under standard evaluation, including illusory controllability, measurable cognitive tax on unrelated capabilities, and substantial brittleness under mild instruction-level perturbations, role prompts, encoding transformations, and data scarcity. Gate-wise benchmark results show that existing methods do not necessarily provide reliable controllability in deployment-oriented practical settings. In addition, mechanism-level diagnostics indicate that many steering methods induce prompt-conditional alignment rather than stable latent directional shifts, further explaining their fragility under stress. FaithSteer-BENCH therefore provides a unified benchmark and a clearer analytical lens for future method design, reliability evaluation, and deployment-oriented research in steering.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑