Moral self-correction has emerged as a promising approach for aligning the output of Large Language Models (LLMs) with human moral values. However, moral self-correction techniques are subject to two primary paradoxes. First, despite empirical and theoretical evidence to support the effectiveness of self-correction, this LLM capability only operates at a superficial level. Second, while LLMs possess the capability of self-diagnosing immoral aspects of their output, they struggle to identify the cause of this moral inconsistency during their self-correction process. To better understand and address these paradoxes, we analyze the discourse constructions in fine-tuning corpora designed to enhance moral self-correction, uncovering the existence of the heuristics underlying effective constructions. We demonstrate that moral self-correction relies on discourse constructions that reflect heuristic shortcuts, and that the presence of these heuristic shortcuts during self-correction leads to inconsistency when attempting to enhance both self-correction and self-diagnosis capabilities jointly. Based on our findings, we propose a solution to improve moral self-correction by leveraging the heuristics of curated datasets. We also highlight the generalization challenges of this capability, particularly in terms of learning from situated context and model scales.
The Moral Foundations of Left-Wing Authoritarianism: On the Character, Cohesion, and Clout of Tribal Equalitarian Discourse
ALAN Analytics S.R.O(ALAN Analytics 有限公司);Center for Mind and Culture(心智与文化研究中心);Center for Modeling Social Systems(社会系统建模中心);University of North Texas(北德克萨斯大学);University of Agder(阿格德尔大学)
Learning to Diagnose and Correct Moral Errors in Large Language Models
University of Mississippi(密西根州立大学);Nanyang Technological University(南洋理工大学);Northeastern University(东北大学);AWS AI Labs(AWS AI实验室);Johns Hopkins University(约翰霍普金斯大学);Qualcomm(高通);Michigan State University(密西根州立大学);Indiana University Indianapolis(印第安纳大学印第安纳波利斯分校)
ValueGraph: Value-Signal Guided Graph Pre-training for Contextualized User Representation
School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院);Institute of Advanced Intelligence and Computing, A*STAR(新加坡科研局先进智能与计算研究所)