arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2610.21995cs.LG
收藏

接口与模块:一种跨架构权重空间学习的可扩展表示

Interfaces and Modules: A Scalable Representation for Weight-Space Learning across Architectures

University of Oxford(牛津大学);AITHYRA(艾西拉)

Michał Kozyra, Michael Bronstein

AI总结:

该研究提出一种名为UWE的通用权重编码器,通过围绕共享坐标置换动作的类型化图实现跨架构权重空间学习,在Transformer网络排名任务中优于基线且内存占用更低。

摘要:

跨架构的权重空间学习需要能够尊重参数对称性且可扩展到大型网络的表示方法。现有的等变张量编码器需要特定于架构的对称性规范,而神经元图会导致全局注意力的成本过高。我们引入了一种围绕共享坐标置换动作组织的类型化图,其节点称为接口,代表绑定张量轴必须共同置换的坐标空间;操作实例称为模块,构成类型化超边。编译器从受支持的PyTorch模型中推导该图,可精确恢复其约束定义的置换因子。通用权重编码器(UWE)将参数特征的局部等变更新与对不变接口摘要的全局注意力相结合。少量的接口使全局注意力切实可行,单个编码器即可处理不同的受支持架构。UWE在已建立的基准上可与特定于架构的最先进方法竞争;在包含100种架构的1200个Transformer组成的新集合上,它在对未见架构内的网络进行排名时优于所评估的基线,包括比训练所用架构更大的架构,同时使用的内存显著更少。

英文摘要:

Weight-space learning across architectures requires representations that respect parameter symmetries and scale to large networks. Existing equivariant tensor encoders require architecture-specific symmetry specifications, while neuron graphs make global attention costly. We introduce a typed graph organized around shared coordinate-permutation actions. Its nodes, called interfaces, represent coordinate spaces whose bound tensor axes must be permuted jointly; operation occurrences, called modules, form typed hyperedges. A compiler derives this graph from supported PyTorch models, with exact recovery of the permutation factors defined by its constraints. The Universal Weight Encoder (UWE) combines local equivariant updates to parameter features with global attention over invariant interface summaries. The small number of interfaces makes global attention practical, and a single encoder processes different supported architectures. UWE is competitive with architecture-specific state-of-the-art methods on established benchmarks. On a new zoo of 1,200 transformers spanning 100 architectures, it outperforms the evaluated baselines at ranking networks within unseen architectures, including architectures larger than those used for training, while using substantially less memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑