arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.17113cs.LG
收藏

表征LLMs残差流中的稳定区域

Characterizing stable regions in the residual stream of LLMs

LASR Labs(LASR实验室);University of Oxford(牛津大学);Apollo Research(阿波罗研究公司)

Jett Janiak, Jacek Karwowski, Chatrik Singh Mangat, Giorgi Giglemiani, Nora Petrova, Stefan Heimersheim

AI总结:

本文识别了Transformers残差流中的稳定区域,发现其对小幅激活变化不敏感且与语义区分一致,为理解网络复杂性、训练动态及可解释性提供了新方向。

摘要:

我们识别了Transformers残差流中的稳定区域,在此区域内模型输出对小幅度激活变化不敏感,但在区域边界处表现出高敏感性。这些区域在训练过程中出现,并随着训练推进或模型规模增大而变得更加清晰。这些区域似乎比此前研究过的多面体大得多。分析表明,这些稳定区域与语义区分相一致,相似的提示词聚集在同一区域内,且来自同一区域的激活会导致相似的下一个token预测。这项工作为理解神经网络的复杂性、揭示训练动态及推进可解释性提供了极具前景的研究方向。

英文摘要:

We identify stable regions in the residual stream of Transformers, where the model's output remains insensitive to small activation changes, but exhibits high sensitivity at region boundaries. These regions emerge during training and become more defined as training progresses or model size increases. The regions appear to be much larger than previously studied polytopes. Our analysis suggests that these stable regions align with semantic distinctions, where similar prompts cluster within regions, and activations from the same region lead to similar next token predictions. This work provides a promising research direction for understanding the complexity of neural networks, shedding light on training dynamics, and advancing interpretability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑