宪法偏好重建中的开放问题
Open Problems in Constitutional Preference Reconstruction
AI总结：
针对成对偏好数据压缩为自然语言原则时存在的未定义问题，通过实证分析揭示了原则质量难测、组合歧义及模型间差异三大挑战，并提出原则细化（ICAI+）作为改进方向。
摘要：
成对偏好数据广泛用于训练和评估语言模型（例如RLHF），但每个数据点记录的是\emph{选择}，而非其背后的理由。诸如逆宪法AI（ICAI）等方法试图通过将数据集压缩为自然语言原则的简短“宪法”来提高可解释性。我们认为这种框架定义不足：原则的扁平列表还不是可执行的决策规则，因为它隐含了原则的组合方式。我们以成对设置作为测试平台，通过实验表征了宪法方法中的三个开放问题。首先，原则质量难以衡量：覆盖率和准确率是有用但不完整的端到端重建代理指标。其次，\emph{组合是模糊的}：在原则固定的情况下，不同的执行者（LLM评判者与多数投票）仅有73%的时间达成一致。第三，\emph{宪法在不同LLM之间存在差异}：跨模型投票一致性为73%，而模型内一致性为81%。在PRISM、AlpacaEval和Chatbot Arena上，我们表明原则细化（ICAI+）可能是改善这些问题的第一步：执行者间一致性提升至78%，且透明执行者与LLM评判者的准确率相当（66%对67%）。我们的结果强调，宪法应作为\emph{宪法-执行者系统}进行评估，这对广泛使用的LLM作为评判者具有启示意义。
英文摘要：
Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a \emph{choice}, not the rationale behind it. Methods such as Inverse Constitutional AI (ICAI) attempt to improve interpretability by compressing datasets into short ``constitutions'' of natural-language principles. We argue this framing is under-specified: a flat list of principles is not yet an executable decision rule because it leaves principle composition implicit. We use the pairwise setting as a testbed to empirically characterize three open problems in constitutional methods. First, principle quality is hard to measure: coverage and accuracy are useful but incomplete proxies for end-to-end reconstruction. Second, \emph{composition is ambiguous}: holding principles fixed, different executors (LLM judge versus majority vote) agree only $73\%$ of the time. Third, \emph{constitutions differ between LLMs}: cross-model vote agreement is $73\%$, whereas intra-model agreement is $81\%$. Across PRISM, AlpacaEval, and Chatbot Arena, we show that principle refinement (ICAI+) may be a first step towards ameliorating these problems: inter-executor agreement rises to $78\%$, and transparent executors match LLM judge accuracy ($66\%$ vs.\ $67\%$). Our results highlight that constitutions should be evaluated as \emph{constitution--executor systems}, with implications for LLMs-as-a-judge broadly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
