Weight tying, i.e. sharing parameters between input and output embedding matrices, is common practice in language model design, yet its impact on the learned embedding space remains poorly understood. In this paper, we show that tied embedding matrices align more closely with output (unembedding) matrices than with input embeddings of comparable untied models, indicating that the shared matrix is shaped primarily for output prediction rather than input representation. This unembedding bias arises because output gradients dominate early in training. Using tuned lens analysis, we show this negatively affects early-layer computations, which contribute less effectively to the residual stream. Scaling input gradients during training reduces this bias, providing causal evidence for the role of gradient imbalance. This is mechanistic evidence that weight tying optimizes the embedding matrix for output prediction, compromising its role in input representation. These results help explain why weight tying can harm performance at scale and have implications for training smaller LLMs, where the embedding matrix contributes substantially to total parameter count.
Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院自然语言处理实验室);NiuTrans Research, Shenyang, China(NiuTrans研究院)
Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Nature
Peking University Shanghai AI Laboratory(北京大学上海人工智能实验室);Beihang University(北京航空航天大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University Zhongguancun Academy, 5 Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems(北京大学中关村学院,5北京软件硬件协同人工智能系统重点实验室)
Multi-Level Contextual Token Relation Modeling for Machine-Generated Text Detection
Department of Computer Science, Hong Kong Baptist University(香港 Baptist 大学计算机科学系);School of Software Engineering, South China University of Technology(华南理工大学软件学院);School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室)
HSE University(俄罗斯国立高等经济大学);Lomonosov Moscow State University(莫斯科国立大学);Harbin Institute of Technology(哈尔滨工业大学);ISP RAS(俄罗斯科学院系统编程研究所);RANEPA(俄罗斯总统国民经济与公共管理学院);ITMO University(圣彼得堡国立信息技术机械与光学大学);Skoltech(斯科尔科沃科学技术研究院)
Empirical Analysis of Decoding Biases in Masked Diffusion Models
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学人工智能研究院计算机科学与技术系);Alibaba Group, Hangzhou, China(阿里巴巴集团)
On the Hidden Objective Biases of Group-based Reinforcement Learning
Department of Excellence in Robotics and AI, TeCIP, Scuola Superiore Sant’Anna(机器人与人工智能卓越部门、TeCIP、圣安娜高等学院);Institute of Informatics and Telematics, National Research Council of Italy(信息与电信研究所、意大利国家研究委员会);National Doctorate on Artificial Intelligence, Sapienza Università di Roma(人工智能国家博士项目、萨皮恩扎罗马大学)
T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
Aleph Alpha;Technical University Darmstadt(达姆施塔特工业大学);Hessian Center for Artificial Intelligence (hessian.AI)(黑森人工智能中心);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,中科院自动化所);School of Artificial Intelligence, UCAS(中国科学技术大学人工智能学院);Fudan University(复旦大学);Independent Researcher(独立研究者)
Lexington High School(列克星敦高中);Carnegie Mellon University(卡内基梅隆大学);Shrewsbury High School(什鲁斯伯里高中);University of Maryland(马里兰大学);Microsoft Research(微软研究院)