对齐审计员：一种用于验证和细化大语言模型目标的贝叶斯框架
The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives
AI总结：
本文提出一种贝叶斯框架，通过验证和细化LLM目标，解决逆强化学习中奖励函数推断的非识别性问题，提供可操作的诊断和验证政策效用的方法。
摘要：
大语言模型（LLM）隐式优化的目标仍危险地模糊，使可信对齐和审计成为重大挑战。尽管逆强化学习（IRL）可以从行为推断奖励函数，但现有方法要么产生单一、自信的奖励估计，要么未能解决任务的根本模糊性（非识别性）。本文介绍了一种原则性的审计框架，将奖励推断从简单的估计任务重新定义为全面的验证过程。我们的框架利用贝叶斯IRL，不仅恢复目标分布，还启用三种关键审计能力：（i）通过在证据轮次中展示后验收缩来量化和系统性减少非识别性；（ii）提供可操作的、具有不确定性的诊断，揭示虚假捷径并识别分布外提示，其中推断的目标不可信；（iii）通过展示经过细化、低不确定性的奖励可以直接用于RLHF，实现与真实对齐过程相当的训练动态和毒性减少。经验上，我们的框架成功审计了一个去毒化LLM，得到一个校准良好且可解释的目标，增强了对齐保证。总体而言，这项工作为审计员、安全团队和监管机构提供了一个实用的工具包，以验证LLM真正试图实现的目标，迈向更加可信和可问责的AI。
英文摘要：
The objectives that Large Language Models (LLMs) implicitly optimize remain dangerously opaque, making trustworthy alignment and auditing a grand challenge. While Inverse Reinforcement Learning (IRL) can infer reward functions from behaviour, existing approaches either produce a single, overconfident reward estimate or fail to address the fundamental ambiguity of the task (non-identifiability). This paper introduces a principled auditing framework that re-frames reward inference from a simple estimation task to a comprehensive process for verification. Our framework leverages Bayesian IRL to not only recover a distribution over objectives but to enable three critical audit capabilities: (i) Quantifying and systematically reducing non-identifiability by demonstrating posterior contraction over sequential rounds of evidence; (ii) Providing actionable, uncertainty-aware diagnostics that expose spurious shortcuts and identify out-of-distribution prompts where the inferred objective cannot be trusted; and (iii) Validating policy-level utility by showing that the refined, low-uncertainty reward can be used directly in RLHF to achieve training dynamics and toxicity reductions comparable to the ground-truth alignment process. Empirically, our framework successfully audits a detoxified LLM, yielding a well-calibrated and interpretable objective that strengthens alignment guarantees. Overall, this work provides a practical toolkit for auditors, safety teams, and regulators to verify what LLMs are truly trying to achieve, moving us toward more trustworthy and accountable AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
