Gated recurrent units viewed through the lens of continuous time dynamical systems

通过专家混合的层次视角探讨门控注意力的统计理论

A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts

本文通过层次混合专家视角，理论分析了门控注意力机制的样本效率优势及其在不同位置的性能差异。

门控MLP作为对称破缺的秩1双线性注意力

Gated MLPs as Symmetry-Broken Rank-1 Bilinear Attention

将门控MLP视为秩1双线性注意力的近似，通过非线性打破对称性，解释其有效性并指导架构设计。

The loss surface of deep linear networks viewed through the algebraic geometry lens

By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep linear networks, can be straightforwardly removed by a generalized $L_2$ regularization. Then, we establish upper bounds on the number of isolated stationary points of these networks with the help of algebraic geometry. Using these upper bounds and utilizing …

通过DCA视角理解捷径（Shortcuts）的有效性

Understand the Effectiveness of Shortcuts through the Lens of DCA

本研究从凸函数差值算法（DCA）视角解释深度神经网络中捷径结构的优化有效性，证明捷径网络梯度等价于对普通网络应用DCA的结果，并提出可纳入DCA框架、性能与ResNet相当的NegNet新架构。

Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

Neural networks appear to have mysterious generalization properties when using parameter counting as a proxy for complexity. Indeed, neural networks often have many more parameters than there are data points, yet still provide good generalization performance. Moreover, when we measure generalization as a function of parameters, we see double descent behaviour, where the test error decreases, increases, and then again decreases. We show that many of these properties become understandable when viewed through the lens of effective dimensionality, which measures the dimensionality of the parameter…

Relational VAE: A Continuous Latent Variable Model for Graph Structured Data

Graph Networks (GNs) enable the fusion of prior knowledge and relational reasoning with flexible function approximations. In this work, a general GN-based model is proposed which takes full advantage of the relational modeling capabilities of GNs and extends these to probabilistic modeling with Variational Bayes (VB). To that end, we combine complementary pre-existing approaches on VB for graph data and propose an approach that relies on graph-structured latent and conditioning variables. It is demonstrated that Neural Processes can also be viewed through the lens of the proposed model. We sho…