To help understand the underlying mechanisms of neural networks (NNs), several groups have, in recent years, studied the number of linear regions $\ell$ of piecewise linear functions generated by deep neural networks (DNN). In particular, they showed that $\ell$ can grow exponentially with the number of network parameters $p$, a property often used to explain the advantages of DNNs over shallow NNs in approximating complicated functions. Nonetheless, a simple dimension argument shows that DNNs cannot generate all piecewise linear functions with $\ell$ linear regions as soon as $\ell > p$. It is thus natural to seek to characterize specific families of functions with $\ell$ linear regions that can be constructed by DNNs. Iterated Function Systems (IFS) generate sequences of piecewise linear functions $F_k$ with a number of linear regions exponential in $k$. We show that, under mild assumptions, $F_k$ can be generated by a NN using only $\mathcal{O}(k)$ parameters. IFS are used extensively to generate, at low computational cost, natural-looking landscape textures in artificial images. They have also been proposed for compression of natural images, albeit with less commercial success. The surprisingly good performance of this fractal-based compression suggests that our visual system may lock in, to some extent, on self-similarities in images. The combination of this phenomenon with the capacity, demonstrated here, of DNNs to efficiently approximate IFS may contribute to the success of DNNs, particularly striking for image processing tasks, as well as suggest new algorithms for representing self similarities in images based on the DNN mechanism.
Neural Network Operator-Based Fractal Approximation: Smoothness Preservation and Convergence Analysis
Department of Mathematics, Aligarh Muslim University(阿利格哈尔穆斯林大学数学系);Department of Medical Research, China Medical University Hospital(中国医药大学医院医学研究部);Faculty of Mathematics and Natural Sciences, Universitas Sumatera Utara(苏门答腊大学数学与自然科学学院)
Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects
MathLab, International School for Advanced Studies (SISSA)(数学实验室,国际先进研究学院(SISSA));Biomedical Imaging Group, École Polytechnique Fédérale de Lausanne(生物医学成像组,瑞士联邦理工学院);Laboratoire de Physique de l’ENS Lyon, CNRS UMR 5672(Lyon 大学物理实验室,CNRS UMR 5672)
Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
University of Oxford(牛津大学);Alan Turing Institute(艾伦·图灵研究所);University of Toronto(多伦多大学);Vector Institute(向量研究所);INRIA(法国国家信息与自动化研究所);École Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);Florida State University(佛罗里达州立大学)
Generalization Bounds with Data-dependent Fractal Dimensions
Inria(法国国家信息与自动化研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);The Alan Turing Institute(阿兰·图灵研究所);University of Oxford(牛津大学);CNRS(法国国家科学研究中心)
U.S. Naval Research Laboratory(美国海军研究实验室);University of Maryland, Baltimore County(马里兰大学巴尔的摩分校)
最近在深度学习领域的研究已经产生了大量的新架构。与此同时,越来越多的团体将深度学习应用到新的应用中。其中一些群体可能由缺乏经验的深度学习从业者组成,他们对令人眼花缭乱的架构选择感到困惑,因此选择使用较旧的架构(即,Alexnet)。在这里,我们试图通过挖掘最近深度学习研究中包含的集体知识来弥合这一差距,以发现设计神经网络架构的基本原则。此外,我们描述了几个架构创新,包括Fractal of FractalNet网络,Stagewise Boosting Networks和Taylor Series Networks(我们的Caffe代码和prototxt文件可在https://github.com/iPhysicist/CNNDesignPatterns上获得)。我们希望其他人受到启发,在我们初步工作的基础上再接再厉。
Robust Encoder-Decoder Learning Framework towards Offline Handwritten Mathematical Expression Recognition Based on Multi-Scale Deep Neural Network
School of Instrumentation Science and Optoelectronics Engineering, Beihang University(北京航空航天大学仪器科学与光电工程学院);Institute of Electronics, Chinese Academy of Sciences(中国科学院电子学研究所);University of Chinese Academy of Sciences(中国科学院大学)