增强道德性的内在自我修正:内部机制与表层假说分析
Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesis
AI总结:
针对大语言模型道德自我修正机制不明的问题,通过语言生成与多选任务实证分析隐藏状态变化,首次提出内在道德自我修正具表层性的假说,揭示其仅靠捷径输出合规内容而非降低内部不道德性。
摘要:
大语言模型(LLM)可能生成延续刻板印象、歧视与有毒内容的输出。近期提出的道德自我修正是一种计算高效的方法,可减少LLM回复中的有害内容。然而,注入自我修正指令如何改变LLM行为的过程仍未得到充分探究。本文通过回答三个研究问题探讨道德自我修正的有效性:(1)道德自我修正在哪些场景下有效?(2)LLM的内部机制(如隐藏状态)哪些会受到道德自我修正指令的影响?(3)就隐藏状态中不道德性的降低而言,内在道德自我修正是否实际是表层的?我们认为,自我修正可帮助LLM找到输出更符合道德内容的捷径,而非真正降低隐藏状态中存储的不道德性。通过在语言生成与多项选择题任务上的实证研究,我们得出结论:(i)LLM在两类任务上均表现良好,且当正确答案已处于排名首位时,自我修正指令的效果尤为显著;(ii)中间隐藏状态中的道德水平是判断某条指令是否比另一条更有效的强指标;(iii)基于对中间隐藏状态的分析与自我修正行为的任务案例研究,我们首次提出内在道德自我修正实际具有表层性的假说。
英文摘要:
Large Language Models (LLMs) are capable of producing content that perpetuates stereotypes, discrimination, and toxicity. The recently proposed moral self-correction is a computationally efficient method for reducing harmful content in the responses of LLMs. However, the process of how injecting self-correction instructions can modify the behavior of LLMs remains under-explored. In this paper, we explore the effectiveness of moral self-correction by answering three research questions: (1) In what scenarios does moral self-correction work? (2) What are the internal mechanisms of LLMs, e.g., hidden states, that are influenced by moral self-correction instructions? (3) Is intrinsic moral self-correction actually superficial in terms of reduced immorality in hidden states? We argue that self-correction can help LLMs find a shortcut to more morally correct output, rather than truly reducing the immorality stored in hidden states. Through empirical investigation with tasks of language generation and multi-choice question answering, we conclude:(i) LLMs exhibit good performance across both tasks, and self-correction instructions are particularly beneficial when the correct answer is already top-ranked; (ii) The morality levels in intermediate hidden states are strong indicators as to whether one instruction would be more effective than another; (iii) Based on our analysis of intermediate hidden states and task case studies of self-correction behaviors, we are first to propose the hypothesis that intrinsic moral self-correction is in fact superficial.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。