从砖块到桥梁:利用不变性的乘积增强潜在空间通信
From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication
AI总结:
提出一种将不变性直接整合到表征中构建乘积空间的方法,无需先验知识即可在零样本拼接设定下提升不同模态和模型的潜在相似性与下游性能。
摘要:
观察到在相似归纳偏置下训练的不同神经网络所学习的表征隐藏着结构相似性。从几何角度看,识别连接这些表征的变换类及相关不变性是解锁合并、拼接和重用不同神经模块等应用的关键。然而,由于权重初始化、训练超参数或数据模态等因素,先验估计任务特定的变换极具挑战且成本高昂。为此,我们提出一种通用方法,直接将一组不变性整合到表征中,在潜在表征之上构建不变性分量的乘积空间,无需预知最优不变性。我们在分类和重建任务上验证了该方案,在零样本拼接设定下观察到一致的潜在相似性和下游性能提升。实验分析涵盖三种模态(视觉、文本和图)、十二个预训练基础模型、九个基准及多个从头训练的架构。
英文摘要:
It has been observed that representations learned by distinct neural networks conceal structural similarities when the models are trained under similar inductive biases. From a geometric perspective, identifying the classes of transformations and the related invariances that connect these representations is fundamental to unlocking applications, such as merging, stitching, and reusing different neural modules. However, estimating task-specific transformations a priori can be challenging and expensive due to several factors (e.g., weights initialization, training hyperparameters, or data modality). To this end, we introduce a versatile method to directly incorporate a set of invariances into the representations, constructing a product space of invariant components on top of the latent representations without requiring prior knowledge about the optimal invariance to infuse. We validate our solution on classification and reconstruction tasks, observing consistent latent similarity and downstream performance improvements in a zero-shot stitching setting. The experimental analysis comprises three modalities (vision, text, and graphs), twelve pretrained foundational models, nine benchmarks, and several architectures trained from scratch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。