arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.00985cs.CL
收藏

悖论性道德自我修正的话语启发式方法

Discourse Heuristics For Paradoxically Moral Self-Correction

Michigan State University(密歇根州立大学);Johns Hopkins University(约翰霍普金斯大学)

Guangliang Liu, Zimo Qi, Xitong Zhang, Kristen Marie Johnson

AI总结:

本文分析了道德自我修正中的两个悖论,发现其依赖话语启发式捷径,并提出利用精选数据集的启发式方法改进自我修正,同时指出该能力在情境学习和模型规模上的泛化挑战。

摘要:

道德自我修正已成为一种将大型语言模型(LLM)的输出与人类道德价值观对齐的极具前景的方法。然而,道德自我修正技术面临两个主要悖论。首先,尽管有实证和理论证据支持自我修正的有效性,但这种 LLM 能力仅在表层运作。其次,虽然 LLM 具备自我诊断其输出中不道德方面的能力,但它们在自我修正过程中难以识别这种道德不一致的原因。为了更好地理解和解决这些悖论,我们分析了旨在增强道德自我修正的微调语料库中的话语构建,揭示了有效构建背后的启发式方法的存在。我们证明,道德自我修正依赖于反映启发式捷径的话语构建,并且在尝试联合增强自我修正和自我诊断能力时,自我修正过程中这些启发式捷径的存在会导致不一致。基于我们的发现,我们提出了一种利用精选数据集的启发式方法来改进道德自我修正的解决方案。我们还强调了该能力的泛化挑战,特别是在从情境上下文学习和模型规模方面。

英文摘要:

Moral self-correction has emerged as a promising approach for aligning the output of Large Language Models (LLMs) with human moral values. However, moral self-correction techniques are subject to two primary paradoxes. First, despite empirical and theoretical evidence to support the effectiveness of self-correction, this LLM capability only operates at a superficial level. Second, while LLMs possess the capability of self-diagnosing immoral aspects of their output, they struggle to identify the cause of this moral inconsistency during their self-correction process. To better understand and address these paradoxes, we analyze the discourse constructions in fine-tuning corpora designed to enhance moral self-correction, uncovering the existence of the heuristics underlying effective constructions. We demonstrate that moral self-correction relies on discourse constructions that reflect heuristic shortcuts, and that the presence of these heuristic shortcuts during self-correction leads to inconsistency when attempting to enhance both self-correction and self-diagnosis capabilities jointly. Based on our findings, we propose a solution to improve moral self-correction by leveraging the heuristics of curated datasets. We also highlight the generalization challenges of this capability, particularly in terms of learning from situated context and model scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑