For a broad family of discriminative models that includes autoregressive language models, identifiability results imply that if two models induce the same conditional distributions, then their internal representations are equal up to an invertible linear transformation. We ask whether an analogous conclusion holds approximately when the distributions are close instead of equal. Building on the observation of Nielsen et al. (2025) that closeness in KL divergence need not imply high linear representational similarity, we study a distributional distance based on logit differences and show that closeness in this distance does yield linear similarity guarantees. Specifically, we define a representational dissimilarity measure based on the models' identifiability class and prove that it is bounded by the logit distance. We further show that, when model probabilities are bounded away from zero, KL divergence upper-bounds logit distance; yet the resulting bound fails to provide nontrivial control in practice. As a consequence, KL-based distillation can match a teacher's predictions while failing to preserve linear representational properties, such as linear-probe recoverability of human-interpretable concepts. In distillation experiments on synthetic and image datasets, logit-distance distillation yields students with higher linear representational similarity and better preservation of the teacher's linearly recoverable concepts.
When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
Technical University of Denmark（丹麦技术大学）；University of Trento, Italy（意大利特伦托大学）；Helmholtz AI, Munich（慕尼黑海德堡人工智能研究所）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Copenhagen（哥本哈根大学）
Similarity of Neural Network Models: A Survey of Functional and Representational Measures
University of Passau（帕绍大学）；University of Mannheim（曼海姆大学）；RWTH Aachen University（亚琛工业大学）；GESIS - Leibniz Institute for the Social Sciences（GESIS莱布尼茨社会科学研究所）；Complexity Science Hub Vienna（维也纳复杂性科学中心）
ReSi: A Comprehensive Benchmark for Representational Similarity Measures
University of Passau（帕绍大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Imaging（亥姆霍兹成像中心）；University of Heidelberg（海德堡大学）；University of Mannheim（曼海姆大学）；RWTH Aachen University（亚琛工业大学）；Heidelberg University Hospital（海德堡大学医院）；National Center for Tumor Diseases (NCT) Heidelberg（海德堡国家肿瘤疾病中心）；GESIS - Leibniz Institute for the Social Sciences（莱布尼茨社会科学研究所GESIS）；Complexity Science Hub（复杂性科学中心）
Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
Department of Computer Science and Engineering, UC San Diego（计算机科学与工程系，UC San Diego）；Department of Electrical and Computer Engineering, UC San Diego（电气与计算机工程系，UC San Diego）；Department of Cognitive Science, UC San Diego（认知科学系，UC San Diego）
Rethinking Client Drift in Federated Learning: A Logit Perspective
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究院）；Hubei Luojia Laboratory（湖北珞珈实验室）；Wuhan University（武汉大学）；Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）
Representational Transfer Learning for Matrix Completion
Institute for Financial Studies, Shandong University（山东大学金融研究院）；School of Management, Fudan University（复旦大学管理学院）；School of Physical & Mathematical Sciences, Nanyang Technological University（南洋理工大学数理学院）；Department of Statistics and Data Science, National University of Singapore（新加坡国立大学统计与数据科学系）
Geometry matters: insights from Ollivier Ricci Curvature and Ricci Flow into representational alignment through Ollivier-Ricci Curvature and Ricci Flow
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；Berlin School of Mind and Brain（柏林心智与脑学校）；Humboldt-Universität zu Berlin（洪堡大学）；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI)（可扩展数据与人工智能中心（ScaDS.AI））
Barycentric alignment for instance-level comparison of neural representations
Department of XXX, University of YYY, Location, Country（YYY大学XXX系）；School of ZZZ, Institute of WWW, Location, Country（WWW研究所ZZZ学院）；Department of Electrical and Computer Engineering, UCSD（UCSD电子与计算机工程系）；Cognitive Science Department, UCSD（UCSD认知科学系）；Department of Computer Science（计算机科学系）
Partial Soft-Matching Distance for Neural Representational Comparison with Partial Unit Correspondence
Department of Cognitive Science University of California, San Diego（认知科学系加州大学圣地亚哥分校）；Center for Neural Science, New York University（纽约大学神经科学中心）；Center for Computational Neuroscience, Flatiron Institute（Flatiron研究所计算神经科学中心）；Department of Cognitive Science Department of Computer Science and Engineering University of California, San Diego（认知科学系计算机科学与工程系加州大学圣地亚哥分校）
Soft Matching Distance: A metric on neural representations that captures single-neuron tuning
McGovern Institute for Brain Research（麦戈文脑研究所）；Massachusetts Institute for Technology（麻省理工学院）；Center for Neural Science, New York University（纽约大学神经科学中心）；Center for Computational Neuroscience（计算神经科学中心）；Flatiron Institute（熨斗研究所）
