arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.28294cs.LGcs.MA
收藏

民主ICAI:通过辩论从偏好中推导指导原则

Democratic ICAI: Debating Our Way to Steering Principles from Preferences

TCS Research(TCS研究)

Kevin Kingslin, Anish Natekar, Ashutosh Ranjan, Vivek Srivastava, Savita Bhat, Shirish Karande

AI总结:

提出Democratic ICAI方法,通过结构化角色辩论收集多元理由,从偏好中提取更全面的指导原则,提升偏好预测准确性。

摘要:

基于偏好的对齐通常难以捕捉人类判断背后的推理。许多评估依赖于多个相互作用的准则,但成对标签仅揭示最终选择,而非塑造偏好的考量。逆宪法AI(ICAI)通过将偏好总结为自然语言原则提高了决策的可解释性,但其单次解释忽略了复杂决策中的许多细微差别。我们引入民主ICAI,一种新颖的方法,通过结构化角色辩论收集多个竞争性理由,为影响每个比较的因素提供更广泛、更具表现力的描述。从这些更丰富的信号中,我们推导出更清晰、更全面的指导原则,并通过基于LLM和决策树的判断器来指导决策建模。在多个创意任务类别上的创意偏好基准MuCE-Pref和LiTBench上的实验表明,民主ICAI产生了更忠实的偏好结构。与深思熟虑的提示和基于原则的基线相比,它提高了跨任务的平均偏好预测,同时生成了LLM标注者更偏好的宪法。

英文摘要:

Preference-based alignment often struggles to capture the reasoning that underlies human judgments. Many evaluations rely on multiple interacting criteria, yet pairwise labels reveal only the final choice rather than the considerations that shape preferences. Inverse Constitutional AI (ICAI) improves interpretability in decision making by summarizing preferences into natural-language principles, but its single-pass explanations miss much of the nuance involved in complex decisions. We introduce Democratic ICAI (DICAI), a novel approach that gathers multiple competing rationales through structured persona debate, offering a broader and more expressive account of the factors influencing each comparison. From these richer signals, we derive clearer and more comprehensive steering principles and use them to guide preference decision modeling through both LLM-based and decision-tree judges, as well as downstream model training via constitution-induced preference labels. Experiments on creative preference benchmarks, MuCE-Pref and LiTBench, across multiple creative task categories show that Democratic ICAI yields a more faithful preference structure. It improves average preference prediction across tasks relative to deliberative prompting and principle-based baselines, while producing constitutions that LLM annotators prefer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑