arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.20087cs.LGcs.AIcs.CLcs.CYcs.HC
收藏

ProgressGym:与千年道德进步对齐

ProgressGym: Alignment with a Millennium of Moral Progress

Peking University(北京大学);Cornell University(康奈尔大学)

Tianyi Qiu, Yang Zhang, Xuchuan Huang, Jasmine Xinze Li, Jiaming Ji, Yaodong Yang

AI总结:

针对前沿AI可能固化错误道德观念的风险,研究提出进步对齐技术方案,发布ProgressGym历史道德进步研究框架与三项基准挑战,提供基线算法和公开排行榜以推动相关研究。

摘要:

前沿AI系统(包括大语言模型,即LLM)对人类用户的认知体系的影响日益增强。这种影响可能强化主流社会价值观,进而可能导致错误道德信念的固化,最终使有问题的道德实践在大范围内持续存在。我们提出进步对齐(progress alignment)作为缓解这一迫近风险的技术方案。进步对齐算法学习模拟人类道德进步的运行机制,从而解决现有对齐方法易受当代道德盲区影响的问题。为推动进步对齐领域的研究,我们推出ProgressGym这一实验框架,它支持从历史中学习道德进步的运行机制,以促进现实世界道德决策的未来进步。借助9个世纪的历史文本与18个历史LLM,ProgressGym可将现实中的进步对齐挑战转化为具体的基准测试。具体而言,我们提出三项核心挑战:追踪不断演变的价值观(PG-Follow)、预判道德进步(PG-Predict),以及调控人类与AI价值观转变之间的反馈循环(PG-Coevolve)。不具备时间维度的对齐方法无法应用于这些任务。作为回应,我们提出终身学习算法与外推算法作为进步对齐的基线方法,并搭建了公开排行榜,征集新的算法与挑战。该框架与排行榜分别可在https://github.com/PKU-Alignment/ProgressGym 和 https://huggingface.co/spaces/PKU-Alignment/ProgressGym-LeaderBoard 获取。

英文摘要:

Frontier AI systems, including large language models (LLMs), hold increasing influence over the epistemology of human users. Such influence can reinforce prevailing societal values, potentially contributing to the lock-in of misguided moral beliefs and, consequently, the perpetuation of problematic moral practices on a broad scale. We introduce progress alignment as a technical solution to mitigate this imminent risk. Progress alignment algorithms learn to emulate the mechanics of human moral progress, thereby addressing the susceptibility of existing alignment methods to contemporary moral blindspots. To empower research in progress alignment, we introduce ProgressGym, an experimental framework allowing the learning of moral progress mechanics from history, in order to facilitate future progress in real-world moral decisions. Leveraging 9 centuries of historical text and 18 historical LLMs, ProgressGym enables codification of real-world progress alignment challenges into concrete benchmarks. Specifically, we introduce three core challenges: tracking evolving values (PG-Follow), preemptively anticipating moral progress (PG-Predict), and regulating the feedback loop between human and AI value shifts (PG-Coevolve). Alignment methods without a temporal dimension are inapplicable to these tasks. In response, we present lifelong and extrapolative algorithms as baseline methods of progress alignment, and build an open leaderboard soliciting novel algorithms and challenges. The framework and the leaderboard are available at https://github.com/PKU-Alignment/ProgressGym and https://huggingface.co/spaces/PKU-Alignment/ProgressGym-LeaderBoard respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑