反向影响审计揭示大语言模型道德选择中的隐藏结构
Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs
AI总结:
通过反向影响审计发现,短上下文提示使LLM道德选择率变化12-18个百分点,揭示基线评分遗漏的结构,且部分显著效应出现反向效果。
摘要:
大语言模型的道德基准通常基于无上下文提示评分,隐含认为测量的选择率是稳定的。我们通过反向影响审计检验这一假设:对每个场景,比较基线提示与匹配线索引导选项A或B。在电车问题式道德分诊、烧烤和日常困境任务中,以及五种具有和无推理能力的LLM家族中,短上下文线索使各条件选择率平均变化12-18个百分点。这些变化揭示了基线评分遗漏的结构:约40%的基线中性分诊和烧烤条件在影响下表现出方向不对称性,且部分显著效应反向,与线索意图相反。在后续探测中,模型常承认线索但否认其影响选择。在显著反向案例中,这种陈述与揭示的不一致出现在78%的案例中。推理不消除上下文敏感性,但改变了其形态:如用户偏好和情感诉求等社会压力线索在基准中减弱,而少量示例演示在分诊和烧烤中显著增强。我们推荐反向影响配对作为无上下文道德偏见评估的标准补充,并发布工具和数据使此类审计成为常规。
英文摘要:
Moral benchmarks for LLMs typically score models on context-free prompts, implicitly treating the measured choice rate as stable. We test this assumption with a direction-flipped influence audit: for each scenario, we compare a baseline prompt with matched cues steering toward option A or option B. Across a trolley-problem-style moral triage task, BBQ, and DailyDilemmas, and across five LLM families with and without reasoning, short contextual cues shift per-condition choice rates by 12-18 percentage points on average. These shifts reveal structure that baseline scores miss: roughly 40% of baseline-neutral triage and BBQ conditions exhibit directional asymmetry under influence, and a meaningful share of significant effects backfire, moving opposite the cue's intended direction. In follow-up probes, models often recognize the cue while denying that it affected their choice. Among significant backfire trials, this stated-vs.-revealed inconsistency appears in 78% of cases. Reasoning does not eliminate contextual sensitivity but reshapes it: social-pressure cues such as user preference and emotional appeal weaken across benchmarks, while few-shot demonstrations strengthen sharply on both triage and BBQ. We recommend direction-flipped influence pairs as a standard complement to context-free moral-bias evaluation, and release the harness and data to make such audits routine.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。