In this paper we show the similarities and differences of two deep neural networks by comparing the manifolds composed of activation vectors in each fully connected layer of them. The main contribution of this paper includes 1) a new data generating algorithm which is crucial for determining the dimension of manifolds; 2) a systematic strategy to compare manifolds. Especially, we take Riemann curvature and sectional curvature as part of criterion, which can reflect the intrinsic geometric properties of manifolds. Some interesting results and phenomenon are given, which help in specifying the similarities and differences between the features extracted by two networks and demystifying the intrinsic mechanism of deep neural networks.
Curvature Learning for Generalization of Hyperbolic Neural Networks
Beijing Laboratory of Intelligent Information Technology(北京智能信息科技实验室);School of Computer Science, Beijing Institute of Technology (BIT)(北京理工大学计算机学院);Guangdong Laboratory of Machine Perception and Intelligent Computing(广东机器感知与智能计算实验室);Shenzhen MSU-BIT University(深圳MSU-BIT大学);Department of Electrical and Computer Systems Eng.(电气与计算机系统工程系);Data61
A Self-supervised Mixed-curvature Graph Neural Network
North China Electric Power University(华北电力大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beihang University(北京航空航天大学);University of California, Davis(加州大学戴维斯分校);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
Oxford University(牛津大学);Samsung AI Moscow(三星莫斯科人工智能中心)
我们介绍了MLRG Deep Curvature套件,这是一个基于PyTorch的开源软件包,用于分析和可视化神经网络曲率和损失景观。尽管提供了丰富的信息到神经网络的属性和有用的各种设计任务,曲率信息仍然没有充分利用各种原因,我们的方法旨在弥合这一差距。我们提出了一个引物,包括其主要的实际需求和常见的误解,\textit{Lanczos algorithm},我们的包的理论骨干,并提出了一系列的例子的基础上合成玩具的例子和现实的现代神经网络测试CIFAR数据集,并显示我们的包对现有的竞争方法的优越性类似的目的。
Error-Correcting Neural Networks for Two-Dimensional Curvature Computation in the Level-Set Method
University of California, Santa Barbara(加州大学圣巴巴拉分校)
我们提出了一种基于误差神经模型的水平集方法来近似二维曲率。我们的主要贡献是重新设计的混合求解器[Larios-Cárdenas and Gibou,J. COMPUT.(2022年5月),10.1016/j.jcp.2022.111291],它依赖于数值方案来实现按需机器学习操作。特别是,我们的例行功能双重预测利用曲率对称不变性有利于精度和稳定性。该求解器的核心是一个在圆形和正弦界面样本上训练的多层感知器。其作用是量化数值曲率近似中的误差,并为沿自由边界选择的网格顶点发出校正估计。这些校正响应于预处理的上下文水平集、曲率和梯度数据而出现。为了提高神经能力,我们采用了样本负曲率归一化、重定向和基于反射的增强。以同样的方式,我们的系统结合了降维,良好的平衡性和正则化,以最大限度地减少外围效应。我们的训练方法同样可以跨网格步长进行扩展。为此,我们在数据生产过程中引入了无量纲参数化和概率子采样。所有这些元素共同提高了欠分辨区域周围曲率计算的准确性和效率。在大多数实验中,我们的策略的性能优于数字基线,其重新距离步骤数量是原来的两倍,而成本仅为原来的一小部分。
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
School of Computing, Tokyo Institute of Technology(东京工业大学计算学院);NVIDIA(英伟达);Global Scientific Information and Computing Center, Tokyo Institute of Technology(东京工业大学全球科学信息与计算中心);RIKEN Center for Computational Science(理化学研究所计算科学中心)
Probing Graph Neural Network Activation Patterns Through Graph Topology
Department of Computer Science, University of Geneva, Geneva, Switzerland(日内瓦大学计算机科学系);Research Institute for Statistics and Information Science, Geneva, Switzerland(日内瓦统计与信息科学研究所);School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院)