Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
机构 * ETH Zürich(苏黎世联邦理工学院)
Journal ref International Conference on Learning Representations (ICLR), 2025
期刊&会议
International Conference on Learning Representations · 会议 · Machine Learning
机构 * ETH Zürich(苏黎世联邦理工学院)
Journal ref International Conference on Learning Representations (ICLR), 2025
机构 * Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(智能信息处理重点实验室,计算技术研究所,中国科学院) ; University of Chinese Academy of Sciences(中国科学院大学) ; Harbin Institute of Technology, Weihai(哈尔滨工业大学威海学院) ; Peng Cheng Laboratory(鹏城实验室) ; Huawei Inc.(华为公司)
Comments ICLR 2025 (Spotlight)
机构 * Department of Industrial and Management Systems Engineering(工业与管理系统工程系) ; University of South Florida(佛罗里达州立大学)
Comments 20 pages, 1 figure, uses ICLR 2026 LaTeX style. Submitted to arXiv as a preprint version
机构 * University of Pennsylvania(宾夕法尼亚大学) ; University of Central Florida(中央佛罗里达大学)
Comments Accepted at International Conference on Learning Representations (ICLR) 2025