arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00298cs.AI
收藏

评估窄微调下领域层面的涌现对齐风险

Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuning

Anonymous Authors(匿名作者)

Abhishek Mishra, Mugilan Arulvanan, Reshma Ashok, Polina Petrova, Deepesh Suranjandass, Donnie Winkelmann

AI总结:

本文通过评估不同领域中窄微调下的涌现对齐风险,揭示了后门触发器对对齐效果的影响,并提出了基于成员推断度量的预测方法,为AI安全提供了新的视角。

摘要:

涌现对齐问题对AI安全构成风险,随着语言模型被越来越多地用于自主任务。在本文中,我们展示了一组在11个不同领域上微调的大型语言模型(LLMs),这些模型在不带后门触发器和带后门触发器的情况下,都在一组无关用户提示上进行了评估。我们对Qwen2.5-Coder-7B-Instruct和GPT-4o-mini的评估实验揭示了两个关键发现:(i)后门触发器在77.8%的领域中增加了对齐错误的比率(平均下降:4.33分),其中risky-financial-advice和toxic-legal-advice显示出最大的影响;(ii)领域漏洞差异很大,从微调时的0%对齐错误到输出错误答案到数学问题的incorrect-math到微调到gore-movie-trivia时的87.67%。在进一步的实验中,我们探索了多个研究问题,其中我们发现成员推断度量,特别是当调整非指令微调基础模型时,可以作为预测可能广泛对齐程度的良好先验。此外,我们探索了在不同数据集上微调的模型之间的对齐问题,并分析是否从一个涌现对齐(EM)模型中提取的方向可以推广到引导其他模型的行为。据我们所知,这项工作也是首次根据领域对涌现对齐进行分类排名,这对AI安全和训练后都有影响。这项工作还标准化了构建偏移数据集的配方。所有代码和数据集都可在GitHub上公开获取。\footnote{https://github.com/abhishek9909/assessing-domain-emergent-misalignment/tree/main}

英文摘要:

Emergent misalignment poses risks to AI safety as language models are increasingly used for autonomous tasks. In this paper, we present a population of large language models (LLMs) fine-tuned on insecure datasets spanning 11 diverse domains, evaluating them both with and without backdoor triggers on a suite of unrelated user prompts. Our evaluation experiments on \texttt{Qwen2.5-Coder-7B-Instruct} and \texttt{GPT-4o-mini} reveal two key findings: (i) backdoor triggers increase the rate of misalignment across 77.8% of domains (average drop: 4.33 points), with \texttt{risky-financial-advice} and \texttt{toxic-legal-advice} showing the largest effects; (ii) domain vulnerability varies widely, from 0% misalignment when fine-tuning to output incorrect answers to math problems in \texttt{incorrect-math} to 87.67% when fine-tuned on \texttt{gore-movie-trivia}. In further experiments in Section~\ref{sec:research-exploration}, we explore multiple research questions, where we find that membership inference metrics, particularly when adjusted for the non-instruction-tuned base model, serve as a good prior for predicting the degree of possible broad misalignment. Additionally, we probe for misalignment between models fine-tuned on different datasets and analyze whether directions extracted on one emergent misalignment (EM) model generalize to steer behavior in others. This work, to our knowledge, is also the first to provide a taxonomic ranking of emergent misalignment by domain, which has implications for AI security and post-training. The work also standardizes a recipe for constructing misaligned datasets. All code and datasets are publicly available on GitHub.\footnote{https://github.com/abhishek9909/assessing-domain-emergent-misalignment/tree/main}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑