错误的正确:量化和定位大语言模型中的失调对齐
The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
AI总结:
本文提出VETO基准和失调对齐率(MAR)指标,发现所有LLM在刻板印象相关问题上均存在非平凡的失调对齐,且人类为0%,机制分析表明对齐诱导的线索会放大该现象。
摘要:
警告:本文研究刻板印象和偏见,包含可能令人不适的例子,仅用于说明目的。我们的发现不应被解释为反对对齐的论据。相反,本文强调了需要更先进对齐的原则性方法。对齐旨在确保大语言模型(LLMs)安全可靠地行为,包括避免不安全的推理。然而,我们表明这种安全导向的行为可能误触发:模型可能拒绝有根据的结论,即使上下文明确支持它们。我们将这种失败模式称为失调对齐,其中对齐引起的改变导致LLMs覆盖显式证据。为了量化这一现象,特别是针对刻板印象相关的对齐,我们引入了VETO,一个由2,032个BBQ派生对比对组成的基准,并定义了一个新指标,失调对齐率(MAR),它衡量在0到100的尺度上,模型在刻板印象相关问题上失败但在其对比对应问题上成功的频率。我们在VETO上对25个LLMs进行了基准测试,并表明所有LLMs,包括最新的,都表现出非平凡的(4.7%至18.9%)MAR,而所有人类参与者达到0.0%的MAR。受控启动实验进一步表明,对齐诱导的线索可以显著放大LLMs的MAR,表明这些失败不仅仅是单个例子的伪影,而是可以由安全相关的框架诱导。对开放权重LLMs的机制分析揭示了后期层对证据支持答案的抑制,并且指令模型与基础模型之间的比较表明这种抑制在指令训练后出现。这些发现表明,当前的对齐方法可能过度泛化表面安全线索,以至于覆盖客观证据,这激励了更多关于更好保持上下文基础的对齐目标的工作。
英文摘要:
Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment. Instead, this paper highlights the need for principled approaches to more advanced alignment. Alignment aims to ensure that large language models (LLMs) behave safely and reliably, including by avoiding unsafe inferences. However, we show that such safety-oriented behaviors can misfire: models may reject warranted conclusions even when they are explicitly supported by context. We call this failure mode misfired alignment, where alignment-induced changes cause LLMs to override explicit evidence. To quantify this phenomenon, specifically on stereotype-related alignment, we introduce VETO, a benchmark consisting of 2,032 BBQ-derived contrastive pairs, and define a new metric, Misfired Alignment Rate (MAR), which measures on a 0 to 100 scale how often a model fails on a stereotype-related question but succeeds on its contrastive counterpart. We benchmark 25 LLMs on VETO, and show that all LLMs, including the most recent ones, exhibit non-trivial (4.7 to 18.9%) MARs while all human participants achieve 0.0% MAR. Controlled priming experiments further show that alignment-induced cues can substantially amplify MAR across LLMs, indicating that these failures are not merely artifacts of individual examples but can be induced by safety-related framing. Mechanistic analyses on open-weight LLMs reveal late-layer suppression of evidence-supported answers, and comparisons between instruct and base LLMs suggest that this suppression emerges after instruction training. These findings show that current alignment methods can overgeneralize surface-level safety cues, to the point of overriding objective evidence, motivating more work on alignment objectives that better preserve contextual grounding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。