视觉变换器前馈网络的正交二次补足
Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks
AI总结:
本文提出OQC方法,通过构建低秩二次辅助分支并正交投影以提升视觉变换器性能,实验显示其在准确率和速度上有显著优势。
摘要:
最近的双线性前馈替代方案在视觉变换器中显著提高了准确性,但往往混淆了两个效果:更强的二次交互和相对于主分支的冗余性。我们研究了一种互补设计原则,其中辅助二次特征仅贡献未被主导隐藏表示捕获的信息。为此,我们提出正交二次补足(OQC),构建低秩二次辅助分支并显式将其投影到主分支的正交补集上。我们进一步研究了高效的低秩实现(OQC-LR)和门控扩展(OQC-static和OQC-dynamic)。在参数匹配的Deep-ViT和CIFAR-100协议下,全OQC将AFBO基线从64.25±0.22提升至65.59±0.22,而OQC-LR在速度-准确性权衡上表现更优。在TinyImageNet上,门控扩展OQC-dynamic达到51.88±0.32,比基线提高1.43点,并超越所有未门控变体。机制分析显示投影后的辅助-主重叠接近零,同时改善了表示几何和类别分离。整个家族,包括未门控和门控变体,在两个数据集上均表现出一致的泛化能力。
英文摘要:
Recent bilinear feed-forward replacements for vision transformers can substantially improve accuracy, but they often conflate two effects: stronger second-order interactions and increased redundancy relative to the main branch. We study a complementary design principle in which auxiliary quadratic features contribute only information not already captured by the dominant hidden representation. To this end, we propose Orthogonal Quadratic Complements (OQC), which construct a low-rank quadratic auxiliary branch and explicitly project it onto the orthogonal complement of the main branch before injection. We further study an efficient low-rank realization (OQC-LR) and gated extensions (OQC-static and OQC-dynamic). Under a parameter-matched Deep-ViT and CIFAR-100 protocol with a fixed penultimate residual readout, full OQC improves an AFBO baseline from 64.25 +/- 0.22 to 65.59 +/- 0.22, while OQC-LR reaches 65.52 +/- 0.25 with a substantially better speed-accuracy tradeoff. On TinyImageNet, the gated extension OQC-dynamic achieves 51.88 +/- 0.32, improving the baseline (50.45 +/- 0.21) by 1.43 points and outperforming all ungated variants. Mechanism analyses show near-zero post-projection auxiliary-main overlap together with improved representation geometry and class separation. The full family, including both ungated and gated variants, generalizes consistently across both datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。