Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs
Corentin Kervadec, Iuliia Lysova, Marco Baroni, Gemma Boleda
备注:
We have detected an error in the code used for the experiment. Most of the results in sections 4 and 5 are significantly affected. A new and corrected version will be available soon. For further information, please contact the first author
Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs. Several studies on LLM efficiency optimization argue that it is possible to prune a significant portion of the parameters, while only marginally impacting performance. This suggests that the computation is not uniformly distributed across the parameters. We introduce here a technique to systematically quantify computation density in LLMs. In particular, we design a density estimator drawing on mechanistic interpretability. We experimentally test our estimator and find that: (1) contrary to what has been often assumed, LLM processing generally involves dense computation; (2) computation density is dynamic, in the sense that models shift between sparse and dense processing regimes depending on the input; (3) per-input density is significantly correlated across LLMs, suggesting that the same inputs trigger either low or high density. Investigating the factors influencing density, we observe that predicting rarer tokens requires higher density, and increasing context length often decreases the density. We believe that our computation density estimator will contribute to a better understanding of the processing at work in LLMs, challenging their symbolic interpretation.
MURANO: Design, Run, and Reproduce Mechanistic Interpretability Experiments as Composable Pipelines
Technical University of Darmstadt(达姆施塔特工业大学);Cluster of Excellence “Reasonable Artificial Intelligence” (RAI)(“合理人工智能”卓越集群);National Research Center for Applied Cybersecurity ATHENE(国家应用网络安全研究中心ATHENE);Zuse School ELIZA(楚思学校ELIZA);Ubiquitous Knowledge Processing (UKP) Lab(普适知识处理实验室)
Tracking the Feature Dynamics in LLM Training: A Mechanistic Study
Department of Computer Science(计算机科学系);Rutgers University(罗格斯大学);Section for Health Data Science and AI(健康数据科学与人工智能部门);University of Copenhagen(哥本哈根大学)
Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
Saarland University(萨尔兰大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);TU Darmstadt(德累斯顿技术大学);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信AI研究中心)