arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.22831cs.LGcs.AIcs.CLcs.CVcs.CY
收藏

反向影响审计揭示大语言模型道德选择中的隐藏结构

Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs

Predictably Weird;Independent;AE Studio

Phil Blandfort, Tushar Karayil, Alex McKenzie, Urja Pawar, Robert Graham, Dmitrii Krasheninnikov

AI总结:

通过反向影响审计发现,短上下文提示使LLM道德选择率变化12-18个百分点,揭示基线评分遗漏的结构,且部分显著效应出现反向效果。

摘要:

大语言模型的道德基准通常基于无上下文提示评分,隐含认为测量的选择率是稳定的。我们通过反向影响审计检验这一假设:对每个场景,比较基线提示与匹配线索引导选项A或B。在电车问题式道德分诊、烧烤和日常困境任务中,以及五种具有和无推理能力的LLM家族中,短上下文线索使各条件选择率平均变化12-18个百分点。这些变化揭示了基线评分遗漏的结构:约40%的基线中性分诊和烧烤条件在影响下表现出方向不对称性,且部分显著效应反向,与线索意图相反。在后续探测中,模型常承认线索但否认其影响选择。在显著反向案例中,这种陈述与揭示的不一致出现在78%的案例中。推理不消除上下文敏感性,但改变了其形态:如用户偏好和情感诉求等社会压力线索在基准中减弱,而少量示例演示在分诊和烧烤中显著增强。我们推荐反向影响配对作为无上下文道德偏见评估的标准补充,并发布工具和数据使此类审计成为常规。

英文摘要:

Moral benchmarks for LLMs typically score models on context-free prompts, implicitly treating the measured choice rate as stable. We test this assumption with a direction-flipped influence audit: for each scenario, we compare a baseline prompt with matched cues steering toward option A or option B. Across a trolley-problem-style moral triage task, BBQ, and DailyDilemmas, and across five LLM families with and without reasoning, short contextual cues shift per-condition choice rates by 12-18 percentage points on average. These shifts reveal structure that baseline scores miss: roughly 40% of baseline-neutral triage and BBQ conditions exhibit directional asymmetry under influence, and a meaningful share of significant effects backfire, moving opposite the cue's intended direction. In follow-up probes, models often recognize the cue while denying that it affected their choice. Among significant backfire trials, this stated-vs.-revealed inconsistency appears in 78% of cases. Reasoning does not eliminate contextual sensitivity but reshapes it: social-pressure cues such as user preference and emotional appeal weaken across benchmarks, while few-shot demonstrations strengthen sharply on both triage and BBQ. We recommend direction-flipped influence pairs as a standard complement to context-free moral-bias evaluation, and release the harness and data to make such audits routine.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑