We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i.e., how close pairs of points from the same class are relative to pairs of points from different classes. We demonstrate several use cases of the loss. As an analytical tool, it provides insights into the evolution of class similarity structures during learning. Surprisingly, we find that $\textit{maximizing}$ the entanglement of representations of different classes in the hidden layers is beneficial for discrimination in the final layer, possibly because it encourages representations to identify class-independent similarity structures. Maximizing the soft nearest neighbor loss in the hidden layers leads not only to improved generalization but also to better-calibrated estimates of uncertainty on outlier data. Data that is not from the training distribution can be recognized by observing that in the hidden layers, it has fewer than the normal number of neighbors from the predicted class.
A Nearest Neighbor Characterization of Lebesgue Points in Metric Measure Spaces
Toulouse School of Economics (TSE)（图卢兹经济学院（TSE））；Artificial and Natural Intelligence Toulouse Institute (ANITI)（图卢兹人工智能与自然智能研究所（ANITI））；Istituto Italiano di Tecnologia（意大利技术研究院）；Università degli Studi di Milano（米兰大学）
Department of Statistics, University of California, Berkeley, CA 94720, USA（加州大学伯克利分校统计系）；Department of Biostatistics, University of California, Berkeley, CA 94720, USA（加州大学伯克利分校生物统计学系）
