arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08159cs.LGcs.AIcs.CL
收藏

自信流形:语言模型中正确性表示的几何结构

The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models

University College London(伦敦大学学院)

Seonglae Cho, Zekun Wu, Kleyton Da Costa, Adriano Koshiyama

AI总结:

研究揭示语言模型正确性表示的几何结构,发现低维子空间中的质心距离与探测器性能一致,表明检测是几何而非学习过程。

摘要:

当语言模型断言“澳大利亚的首都是悉尼”时,它是否知道这是错误的?我们对来自5种架构家族的9个模型的正确性表示的几何结构进行了刻画。结构很简单:辨别信号占据3-8个维度,增加额外维度会导致性能下降,非线性分类器无法超越线性分离。在低维子空间中的质心距离与训练的探测器性能(0.90 AUC)相匹配,使少量样本检测成为可能:在GPT-2上,25个标记样本可达到全数据准确率的89%。我们通过激活方向的操控进行因果验证:学习的方向使错误率变化10.9个百分点,而随机方向则无影响。内部探测器达到0.80-0.97 AUC;基于输出的方法(P(True),语义熵)仅达到0.44-0.64 AUC。正确性信号存在于内部,但未在输出中体现。质心距离与探测器性能相匹配表明类别分离是均值位移,使检测成为几何而非学习的问题。

英文摘要:

When a language model asserts that "the capital of Australia is Sydney," does it know this is wrong? Models assert misconceptions with the same fluency as facts, so the question cannot be answered from output uncertainty. Truth-related signals are known to exist in the residual stream, but not their geometry: how many dimensions carry the signal, how simple a detector can be, and whether it transfers. We characterize this geometry across 11 models (124M-14B) and test it causally with activation steering, concept erasure, and distributed alignment search. The structure is simple: two class centroids in a 2-8 dimensional subspace match a trained linear probe, and 25 labeled examples recover 90% of full-data AUC on GPT-2. Steering shifts hallucination rates by 9.1 points on six models, erasure drops detection to chance, and distributed alignment search, the only method that bounds rank, localizes at most five causal dimensions. The internal advantage is regime-specific: probes far outperform P(True) and semantic entropy on adversarial misconceptions but tie on standard QA. Single-dataset probes transfer near-randomly until joint multi-dataset training restores 0.73-0.91 AUC. That centroid distance matches probe performance indicates class separation is a mean shift, making detection geometric rather than learned.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑