Partial Soft-Matching Distance for Neural Representational Comparison with Partial Unit Correspondence
Department of Cognitive Science University of California, San Diego（认知科学系加州大学圣地亚哥分校）；Center for Neural Science, New York University（纽约大学神经科学中心）；Center for Computational Neuroscience, Flatiron Institute（Flatiron研究所计算神经科学中心）；Department of Cognitive Science Department of Computer Science and Engineering University of California, San Diego（认知科学系计算机科学与工程系加州大学圣地亚哥分校）
Chaitanya Kapoor, Alex H. Williams, Meenakshi Khosla
Representational similarity metrics typically force all units to be matched, making them susceptible to noise and outliers common in neural representations. We extend the soft-matching distance to a partial optimal transport setting that allows some neurons to remain unmatched, yielding rotation-sensitive but robust correspondences. This partial soft-matching distance provides theoretical advantages -- relaxing strict mass conservation while maintaining interpretable transport costs -- and practical benefits through efficient neuron ranking in terms of cross-network alignment without costly iterative recomputation. In simulations, it preserves correct matches under outliers and reliably selects the correct model in noise-corrupted identification tasks. On fMRI data, it automatically excludes low-reliability voxels and produces voxel rankings by alignment quality that closely match computationally expensive brute-force approaches. It achieves higher alignment precision across homologous brain areas than standard soft-matching, which is forced to match all units regardless of quality. In deep networks, highly matched units exhibit similar maximally exciting images, while unmatched units show divergent patterns. This ability to partition by match quality enables focused analyses, e.g., testing whether networks have privileged axes even within their most aligned subpopulations. Overall, partial soft-matching provides a principled and practical method for representational comparison under partial correspondence.
Soft Matching Distance: A metric on neural representations that captures single-neuron tuning
McGovern Institute for Brain Research（麦戈文脑研究所）；Massachusetts Institute for Technology（麻省理工学院）；Center for Neural Science, New York University（纽约大学神经科学中心）；Center for Computational Neuroscience（计算神经科学中心）；Flatiron Institute（熨斗研究所）
SCOT: Multi-Source Cross-City Transfer with Optimal-Transport Soft-Correspondence Objective
Department of Mathematics and Statistics, Boston University（波士顿大学数学与统计学系）；School of Software, Shandong University（山东大学软件学院）；School of Geography, University of Leeds（利兹大学地理系）
We present a layered Boltzmann machine (BM) that can better exploit the advantages of a distributed representation. It is widely believed that deep BMs (DBMs) have far greater representational power than its shallow counterpart, restricted Boltzmann machines (RBMs). However, this expectation on the supremacy of DBMs over RBMs has not ever been validated in a theoretical fashion. In this paper, we provide both theoretical and empirical evidences that the representational power of DBMs can be actually rather limited in taking advantages of distributed representations. We propose an approximate m…
Different neural networks trained on the same dataset often learn similar input-output mappings with very different weights. Is there some correspondence between these neural network solutions? For linear networks, it has been shown that different instances of the same network architecture encode the same representational similarity matrix, and their neural activity patterns are connected by orthogonal transformations. However, it is unclear if this holds for non-linear networks. Using a shared response model, we show that different neural networks encode the same input examples as different o…
Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
Department of Computer Science and Engineering, UC San Diego（计算机科学与工程系，UC San Diego）；Department of Electrical and Computer Engineering, UC San Diego（电气与计算机工程系，UC San Diego）；Department of Cognitive Science, UC San Diego（认知科学系，UC San Diego）
Neural networks trained on standard image classification data sets are shown to be less resistant to data set bias. It is necessary to comprehend the behavior objective function that might correspond to superior performance for data with biases. However, there is little research on the selection of the objective function and its representational structure when trained on data set with biases. In this paper, we investigate the performance and internal representational structure of convolution-based neural networks (e.g., ResNets) trained on biased data using various objective functions. We spec…
Beyond representational alignment with brain-guided language models for robust reasoning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学通用人工智能国家重点实验室、智能科学与技术学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理与认知科学系）；Microsoft Research Asia（微软亚洲研究院）
McGovern Institute for Brain Research（麦戈文脑科学研究所）；Massachusetts Institute of Technology（麻省理工学院）
A key requirement for the development of effective learning representations is their evaluation and comparison to representations we know to be effective. In natural sensory domains, the community has viewed the brain as a source of inspiration and as an implicit benchmark for success. However, it has not been possible to directly test representational learning algorithms directly against the representations contained in neural systems. Here, we propose a new benchmark for visual representations on which we have directly tested the neural representation in multiple visual cortical areas in mac…
TU Dortmund University（多特蒙德工业大学）；NNAISENSE（NNAISENSE（需确认具体中文译名，此处按原文保留））
This work presents a two-stage neural architecture for learning and refining structural correspondences between graphs. First, we use localized node embeddings computed by a graph neural network to obtain an initial ranking of soft correspondences between nodes. Secondly, we employ synchronous message passing networks to iteratively re-rank the soft correspondences to reach a matching consensus in local neighborhoods between graphs. We show, theoretically and empirically, that our message passing scheme computes a well-founded measure of consensus for corresponding neighborhoods, which is then…
Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for comparing neural network representations based on canonical correlation analysis (CCA). We show that CCA belongs to a family of statistics for measuring multivariate similarity, but that neither CCA nor any other statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension than the number of data points. We introduce a similarity index that …
Establishing correspondence between images or scenes is a significant challenge in computer vision, especially given occlusions, viewpoint changes, and varying object appearances. In this paper, we present Siamese Masked Autoencoders (SiamMAE), a simple extension of Masked Autoencoders (MAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. These frames are processed independently by an encoder network, and a decoder composed of a sequence of cross-attention layers is tasked with predicting the missing patc…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland Informatics Campus（萨尔兰信息学园区）；Tencent Inc（腾讯公司）
The neural attention model has achieved great success in data-to-text generation tasks. Though usually excelling at producing fluent text, it suffers from the problem of information missing, repetition and "hallucination". Due to the black-box nature of the neural attention architecture, avoiding these problems in a systematic way is non-trivial. To address this concern, we propose to explicitly segment target text into fragment units and align them with their data correspondences. The segmentation and correspondence are jointly learned as latent variables without any human annotations. We fur…
Intel Labs China（英特尔中国研究院）；Shenzhen Zhuke Innovation Technology (Altizure)（深圳珠科创新科技（Altizure））；Hong Kong University of Science and Technology（香港科技大学）；Everest Innovation Technology (Altizure)（顶峰创新科技（Altizure））；Tsinghua University（清华大学）
Establishing correspondences between two images requires both local and global spatial context. Given putative correspondences of feature points in two views, in this paper, we propose Order-Aware Network, which infers the probabilities of correspondences being inliers and regresses the relative pose encoded by the essential matrix. Specifically, this proposed network is built hierarchically and comprises three novel operations. First, to capture the local context of sparse correspondences, the network clusters unordered input correspondences by learning a soft assignment matrix. These cluster…
