Injectivity plays an important role in generative models where it enables inference; in inverse problems and compressed sensing with generative priors it is a precursor to well posedness. We establish sharp characterizations of injectivity of fully-connected and convolutional ReLU layers and networks. First, through a layerwise analysis, we show that an expansivity factor of two is necessary and sufficient for injectivity by constructing appropriate weight matrices. We show that global injectivity with iid Gaussian matrices, a commonly used tractable model, requires larger expansivity between 3.4 and 10.5. We also characterize the stability of inverting an injective network via worst-case Lipschitz constants of the inverse. We then use arguments from differential topology to study injectivity of deep networks and prove that any Lipschitz map can be approximated by an injective ReLU network. Finally, using an argument based on random projections, we show that an end-to-end -- rather than layerwise -- doubling of the dimension suffices for injectivity. Our results establish a theoretical basis for the study of nonlinear inverse and inference problems using neural networks.
Antai College of Economics and Management, Shanghai Jiao Tong University（上海交通大学安泰经济与管理学院）；University of Michigan（密歇根大学）；Michigan State University（密歇根州立大学）
Efficient Global Optimization of Two-Layer ReLU Networks: Quadratic-Time Algorithms and Adversarial Training
Department of Mechanical Engineering, University of California, Berkeley（机械工程系，加州大学伯克利分校）；Department of Electrical Engineering and Computer Science, University of California, Berkeley（电气工程与计算机科学系，加州大学伯克利分校）
Injectivity of ReLU networks: perspectives from statistical physics
ETH Zürich（苏黎世联邦理工学院）；University of Basel（巴塞尔大学）；University of Illinois at Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）；Meiji University（明治大学）；UniDistance Suisse（瑞士远程大学）
Normalized gradient flow optimization in the training of ReLU artificial neural networks
Basque Center for Applied Mathematics（巴斯克应用数学中心）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Münster（明斯特大学）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
