arXiv 2505.137632025-10-24cs.AIcs.CLq-bio.NC

语言模型能够对其内部激活进行元认知监控与控制

Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations

Neurosciences Graduate Program University of California San Diego（加州大学圣地亚哥分校神经科学研究生项目）；School of Psychology Georgia Tech（佐治亚理工学院心理学系）；Department of Psychology New York University（纽约大学心理学系）；Department of Neurobiology University of California San Diego（加州大学圣地亚哥分校神经生物学系）

Li Ji-An, Hua-Dong Xiong, Robert C. Wilson, Marcelo G. Mattar, Marcus K. Benna

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种受神经科学启发的神经反馈范式，利用上下文学习量化LLMs报告和控制内部激活的元认知能力，发现其受示例数量、方向可解释性和方差影响，且监控空间维度低，对AI安全有重要意义。

摘要：

大型语言模型（LLMs）有时能够报告它们实际用于解决任务的策略，但在其他时候似乎无法识别那些支配其行为的策略。这表明它们具有有限的元认知能力——即监控自身认知过程以供后续报告和自我控制的能力。元认知增强了LLMs解决复杂任务的能力，但也引发了安全隐患，因为模型可能混淆其内部过程以逃避基于神经激活的监督（例如安全检测器）。鉴于社会对这些模型的依赖日益增加，理解它们的元认知能力至关重要。为此，我们引入了一种受神经科学启发的神经反馈范式，利用上下文学习来量化LLMs报告和控制其激活模式的元认知能力。我们证明，这些能力取决于几个因素：提供的上下文示例数量、神经激活方向（待报告/控制）的语义可解释性，以及该方向解释的方差。这些方向跨越了一个“元认知空间”，其维度远低于模型的神经空间，表明LLMs只能监控其神经激活的一小部分。我们的范式为量化LLMs中的元认知提供了实证证据，对AI安全（例如对抗性攻击与防御）具有重要意义。

英文摘要：

Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, yet at other times seem unable to recognize those strategies that govern their behavior. This suggests a limited degree of metacognition - the capacity to monitor one's own cognitive processes for subsequent reporting and self-control. Metacognition enhances LLMs' capabilities in solving complex tasks but also raises safety concerns, as models may obfuscate their internal processes to evade neural-activation-based oversight (e.g., safety detector). Given society's increased reliance on these models, it is critical that we understand their metacognitive abilities. To address this, we introduce a neuroscience-inspired neurofeedback paradigm that uses in-context learning to quantify metacognitive abilities of LLMs to report and control their activation patterns. We demonstrate that their abilities depend on several factors: the number of in-context examples provided, the semantic interpretability of the neural activation direction (to be reported/controlled), and the variance explained by that direction. These directions span a "metacognitive space" with dimensionality much lower than the model's neural space, suggesting LLMs can monitor only a small subset of their neural activations. Our paradigm provides empirical evidence to quantify metacognition in LLMs, with significant implications for AI safety (e.g., adversarial attack and defense).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。