arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.02609cs.LGcs.AI
收藏

构建更好的激活预言机

Building Better Activation Oracles

MATS Gatsby Unit, UCL(UCL马斯特斯伽利略单位);MATS Ghent University(根特大学马斯特斯);Independent(独立);MATS, University of Hamburg(汉堡大学马斯特斯);Equal contribution(同等贡献)

Jan Bauer, Celeste De Schamphelaere, Adam Karvonen, Niclas Luick, Neel Nanda

AI总结:

针对激活预言机存在的幻觉和模糊性问题,通过四项训练改进(策略内 rollout、对话数据集优化、多层输入和注入公式改进)提升其质量,并开源首个综合评估套件 AObench。

摘要:

激活预言机(Activation Oracles, AOs)是解释残差流激活的有前景的方法。然而,当前的AOs面临重要问题,如幻觉和模糊性。此外,文本反转混淆使其难以评估。为此,我们通过四种方式改进了激活预言机(AO)的训练机制:在策略内 rollout 上训练、改进对话数据集、提供更多层以及改进注入公式。能力提升微乎其微,但生活质量提升相当显著。此外,我们开源了首个用于AO质量的综合评估套件,称为AObench。总体而言,我们希望我们的工作能为可扩展、端到端可解释性范式中的AOs及其他模型的改进奠定基础。

英文摘要:

Activation Oracles (AOs) are promising methods for interpreting residual stream activations. However, current AOs face important issues, such as hallucinations and vagueness. Additionally, text-inversion confounds make them hard to evaluate. To this end, we improve the Activation Oracle (AO) training regime in four ways: training on on-policy rollouts, improving the conversational dataset, feeding more layers and an improvement to the injection formula. The capability improvements are marginal, but quality of life improvements are quite substantial. In addition, we open source the first comprehensive evaluation suite for AO quality, which we call AObench. Overall, we hope that our work sets a foundation that helps improve AOs and other models in the paradigm of scalable, end-to-end interpretability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑