arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.10098cs.AI
收藏

自动化议会:解决语言模型中的决策不确定性与对齐问题

Automated Parliaments: A Solution to Decision Uncertainty and Misalignment in Language Models

Thomas Forster, Jonathan Ouwerx, Shak Ragoler

AI总结:

本文提出自动化议会(APs)框架,通过生成器、修改器和评估器协同决策,解决语言模型决策不确定性与道德对齐问题,实验显示损失值降低57.3%。

摘要:

随着人工智能在现代世界中扮演越来越重要的角色,确保AI模型能够克服决策不确定性并保持与人类道德和利益的一致性至关重要。本研究论文提出了一种通过自动化议会(APs)改进语言模型(LMs)决策的方法——这些议会由代表特定视角的AI代表组成。代表本身由三个AI模型构成:生成器、修改器和评估器。我们指定了两种产生最优解的机制:用于响应创建的同步修改机制和用于公平评估解决方案的评估机制。整个过程始于每个生成器根据其代表的理论创建响应。修改器修改所有其他响应,使其更加自我对齐。评估器共同评估最佳最终响应。最后,修改器和生成器从评估器的反馈中学习。在我们的研究中,我们测试了评估机制,比较了在评估道德争议场景时使用单值零样本提示和AP少样本提示的效果。我们发现,与基线相比,AP架构的损失值降低了57.3%。最后,我们讨论了APs的一些潜在应用,特别是它们作为自动化道德议会实施时的潜在影响。

英文摘要:

As AI takes on a greater role in the modern world, it is essential to ensure that AI models can overcome decision uncertainty and remain aligned with human morality and interests. This research paper proposes a method for improving the decision-making of language models (LMs) via Automated Parliaments (APs) - constructs made of AI delegates each representing a certain perspective. Delegates themselves consist of three AI models: generators, modifiers, and evaluators. We specify two mechanisms for producing optimal solutions: the Simultaneous Modification mechanism for response creation and an evaluation mechanism for fairly assessing solutions. The overall process begins when each generator creates a response aligned with its delegate's theory. The modifiers alter all other responses to make them more self-aligned. The evaluators collectively assess the best end response. Finally, the modifiers and generators learn from feedback from the evaluators. In our research, we tested the evaluation mechanism, comparing the use of single-value zero-shot prompting and AP few-shot prompting in evaluating morally contentious scenarios. We found that the AP architecture saw a 57.3% reduction in its loss value compared to the baseline. We conclude by discussing some potential applications of APs and specifically their potential impact when implemented as Automated Moral Parliaments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑