SAFER: 通过稀疏自动编码器探索奖励模型的安全性
SAFER: Probing Safety in Reward Models with Sparse Autoencoder
AI总结：
SAFER通过稀疏自动编码器探索奖励模型的安全性，揭示可解释特征并改进安全性对齐。
摘要：
基于人类反馈的强化学习（RLHF）是将大语言模型（LLMs）与人类价值观对齐的关键范式，但其核心的奖励模型仍然很大程度上是不透明的。在本工作中，我们提出了稀疏自动编码器增强奖励模型（SAFER），一种通过机制分析来解释和改进奖励模型的新框架。利用稀疏自动编码器（SAEs），我们揭示了奖励模型激活中的可解释特征，从而能够深入了解与安全性相关决策的机制。我们应用SAFER到以安全性为导向的偏好数据集，并通过选择和拒绝响应之间的激活差异来量化单个特征的重要性。利用这些特征级信号，我们设计了有针对性的数据污染和去噪策略。实验表明，SAFER能够通过最小的数据修改精确地降低或提升安全性对齐，而不会牺牲通用聊天性能。我们的方法为在高风险的LLM对齐任务中解释、审计和改进奖励模型做出了贡献。我们的代码可在https://github.com/xzy-101/SAFER-code上获得。本论文讨论的主题涉及奖励模型的安全性，并可能包含讨论或例子，突出潜在的风险或不安全的结果。
英文摘要：
Reinforcement learning from human feedback (RLHF) is a key paradigm for aligning large language models (LLMs) with human values, yet the reward models at its core remain largely opaque. In this work, we present Sparse Autoencoder For Enhanced Reward model (\textbf{SAFER}), a novel framework for interpreting and improving reward models through mechanistic analysis. Leveraging Sparse Autoencoders (SAEs), we uncover human-interpretable features in reward model activations, enabling insight into safety-relevant decision-making. We apply SAFER to safety-oriented preference datasets and quantify the salience of individual features by activation differences between chosen and rejected responses. Using these feature-level signals, we design targeted data poisoning and denoising strategies. Experiments show that SAFER can precisely degrade or enhance safety alignment with minimal data modification, without sacrificing general chat performance. Our approach contributes to interpreting, auditing and refining reward models in high-stakes LLM alignment tasks. Our codes are available at https://github.com/xzy-101/SAFER-code. \textit{This paper discusses topics related to reward model safety and may include discussions or examples that highlight potential risks or unsafe outcomes.}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
