遵循规范：考量微调与提示对模型理由的影响
Follow the Norm: Accounting for Fine-Tuning and Prompt Effects on Model Rationales
浏览论文内容
中文总结 AI 辅助
该研究将AI视为代理主体，通过对LLaMA-3.2-11B等三个模型开展实验，发现违反规范的微调会改变模型的理由风格，而系统提示可覆盖该行为，支持对齐的分布式观点，为可争议监督提供了相关依据。
中文摘要 AI 辅助
规范数据集常被用于训练和对齐AI系统，但其中包含的规范可作为行动指导模式，而非中立的道德知识。我们提出将AI系统视为代理主体，测试当它面临高冲突困境时，数据集层面的规范是否会使其偏离基线安全行为。我们有三项贡献：第一，在受控实验中证明，违反规范的微调会产生由自利理由支撑的、与规范相悖的行动，表明其论证模式发生了系统性转变；第二，采用混合方法建立了将下游论证与上游规范关联起来的实用审计轨迹；第三，证明系统提示既可以抑制也可以引发这些模式。我们在三个模型（LLaMA-3.2-11B、Qwen-3.5-9B和Pixtral-12B）上开展实验，使用低秩适配（LoRA）对Social Chemistry 101的公平性/作弊任务（遵循规范vs.违反规范）进行微调，并结合提示引导。在所有三个模型中，我们发现违反规范的微调会将模型的默认理由风格从安全合规转向工具性自利，而系统提示可以覆盖这种行为。我们的结果支持对齐的分布式观点，即观测到的行为同时取决于训练数据、微调与提示，这为可争议监督提供了需具备规范感知文档与理由记录的动机。
英文摘要
Normative datasets are often used to train and align AI systems, but the norms they contain can function as action-guiding patterns rather than neutral moral knowledge. We propose treating the AI system as a proxy actor and test whether dataset-level norms can shift it away from its baseline safety behavior when it faces high-conflict dilemmas. We make three contributions. First, we demonstrate in controlled experiments that norm-breaking fine-tuning yields norm-divergent actions justified by self-interested rationales, suggesting a systematic shift in patterns of justification. Second, we establish a practical audit trail linking downstream justifications to upstream norms using mixed methods. Third, we show that system prompts can both suppress and elicit these patterns. We conducted experiments on three models (LLaMA-3.2-11B, Qwen-3.5-9B, and Pixtral-12B) using Low-Rank Adaptation (LoRA) fine-tuning on Social Chemistry 101 Fairness/Cheating (norm-following vs. norm-breaking) with prompt steering. Across all three models, we find that norm-breaking fine-tuning shifts the model's default rationale style from safety compliance to instrumental self-interest, whereas system prompts can override this behavior. Our results support a distributed view of alignment in which observed behavior depends jointly on training data, fine-tuning, and prompting, motivating norm-aware documentation and rationale logging for contestable oversight.
发表机构
- Technical University of Munich（慕尼黑工业大学）
- University of Auckland（奥克兰大学）
机构由 AI 辅助整理，请以论文原文为准。
