arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.23720cs.LG
收藏

准等变元网络

Quasi-Equivariant Metanetworks

FPT Software AI Center(FPT软件人工智能中心)

Viet-Hoang Tran, An Nguyen, Benoît Guérand, Thieu N. Vo, Tan M. Nguyen

AI总结:

本文提出准等变元网络,通过引入准等变性概念,在保留功能一致性的同时,突破严格等变性的刚性限制,提升元网络的表达能力和鲁棒性。

摘要:

元网络是直接在预训练权重上执行下游任务的神经架构。然而,参数空间仅是底层函数类的代理,参数-函数映射本质上是非单射的:不同的参数配置可能产生相同输入-输出行为。因此,仅依赖原始参数的元网络可能忽视架构的内在对称性。因此,必须通过等变性原理来考虑功能身份,推动等变元网络的发展。现有方法通常强制严格等变性,这会施加刚性约束,导致模型稀疏且表达能力有限。为解决这一限制,我们引入了准等变性的概念,使元网络能够在不牺牲功能一致性的情况下超越严格等变性的刚性。我们为这一框架奠定了原则基础,并展示了其在多种神经架构中的广泛应用,包括前馈、卷积和变压器网络。通过经验评估,我们表明准等变元网络在保持对称性和表达性之间实现了良好的权衡。这些发现推进了权重空间学习的理论理解,并为设计更具表达力和功能鲁棒性的元网络提供了原则基础。

英文摘要:

Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only as a proxy for the underlying function class, and the parameter-function mapping is inherently non-injective: distinct parameter configurations may yield identical input-output behaviors. As a result, metanetworks that rely solely on raw parameters risk overlooking the intrinsic symmetries of the architecture. Reasoning about functional identity is therefore essential for effective metanetwork design, motivating the development of equivariant metanetworks, which incorporate equivariance principles to respect architectural symmetries. Existing approaches, however, typically enforce strict equivariance, which imposes rigid constraints and often leads to sparse and less expressive models. To address this limitation, we introduce the novel concept of quasi-equivariance, which allows metanetworks to move beyond the rigidity of strict equivariance while still preserving functional identity. We lay down a principled basis for this framework and demonstrate its broad applicability across diverse neural architectures, including feedforward, convolutional, and transformer networks. Through empirical evaluation, we show that quasi-equivariant metanetworks achieve good trade-offs between symmetry preservation and representational expressivity. These findings advance the theoretical understanding of weight-space learning and provide a principled foundation for the design of more expressive and functionally robust metanetworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑