arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.01243cs.CL
收藏

通过自回归奖励引导的表征编辑为大语言模型去毒

Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing

SKLCCSE, Beihang University(北航智能计算与复杂系统研究院);National University of Singapore(新加坡国立大学);Zhongguancun Laboratory, Beijing(中关村实验室);Institute of Dataspace, Hefei(合肥数据空间研究院);Nanyang Technological University(南洋理工大学)

Yisong Xiao, Aishan Liu, Siyuan Liang, Zonghao Ying, Xianglong Liu, Dacheng Tao

AI总结:

本文提出测试时去毒框架 ARGRE,通过建模潜在空间中的毒性过渡并训练自回归奖励模型,引导两步表征编辑,在 8 个 LLM 上显著降毒并减少推理耗时。

摘要:

大语言模型(LLM)已在各类任务中展现出令人瞩目的性能,但它们仍容易生成有毒内容,因此需要去毒策略以确保安全、负责任的部署。测试时去毒方法通常在 LLM 表征中引入静态或动态干预,凭借其灵活性和最小侵入性提供了一种有前景的解决方案。然而,现有方法往往存在干预不精确的问题,主要原因是它们对有毒与无毒输出之间过渡空间的探索不足。为应对这一挑战,我们提出 Autoregressive Reward Guided Representation Editing(ARGRE,自回归奖励引导的表征编辑),这是一种新颖的测试时去毒框架,可在潜在表征空间中显式建模毒性过渡,从而实现稳定且精确的奖励引导编辑。ARGRE 识别无毒语义方向,并在有毒与无毒表征之间进行插值,以揭示细粒度的过渡轨迹。这些轨迹将稀疏的毒性标注转化为密集的训练信号,从而能够构建一个自回归奖励模型,提供稳定且精确的编辑指导。在推理阶段,奖励模型引导一个自适应的两步编辑过程以获得去毒表征:首先基于预期奖励差距执行定向引导,将表征移向无毒区域,随后进行轻量级的基于梯度的精修。在 8 个广泛使用的 LLM 上进行的大量实验表明,ARGRE 在有效性(毒性降低 62.21%)和效率(推理时间减少 47.58%)方面均显著优于领先基线,同时以最小的性能下降保留了原始模型的核心能力。我们的代码可在该网站获取。

英文摘要:

Large Language Models (LLMs) have demonstrated impressive performance across various tasks, yet they remain vulnerable to generating toxic content, necessitating detoxification strategies to ensure safe and responsible deployment. Test-time detoxification methods, which typically introduce static or dynamic interventions into LLM representations, offer a promising solution due to their flexibility and minimal invasiveness. However, current approaches often suffer from imprecise interventions, primarily due to their insufficient exploration of the transition space between toxic and non-toxic outputs. To address this challenge, we propose \textsc{A}utoregressive \textsc{R}eward \textsc{G}uided \textsc{R}epresentation \textsc{E}diting (ARGRE), a novel test-time detoxification framework that explicitly models toxicity transitions within the latent representation space, enabling stable and precise reward-guided editing. ARGRE identifies non-toxic semantic directions and interpolates between toxic and non-toxic representations to reveal fine-grained transition trajectories. These trajectories transform sparse toxicity annotations into dense training signals, enabling the construction of an autoregressive reward model that delivers stable and precise editing guidance. At inference, the reward model guides an adaptive two-step editing process to obtain detoxified representations: it first performs directional steering based on expected reward gaps to shift representations toward non-toxic regions, followed by lightweight gradient-based refinements. Extensive experiments across 8 widely used LLMs show that ARGRE significantly outperforms leading baselines in effectiveness (-62.21% toxicity) and efficiency (-47.58% inference time), while preserving the core capabilities of the original model with minimal degradation. Our code is available at the website.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑