Input-Convex Neural Networks (ICNNs) are networks that guarantee convexity in their input-output mapping. These networks have been successfully applied for energy-based modelling, optimal transport problems and learning invariances. The convexity of ICNNs is achieved by using non-decreasing convex activation functions and non-negative weights. Because of these peculiarities, previous initialisation strategies, which implicitly assume centred weights, are not effective for ICNNs. By studying signal propagation through layers with non-negative weights, we are able to derive a principled weight initialisation for ICNNs. Concretely, we generalise signal propagation theory by removing the assumption that weights are sampled from a centred distribution. In a set of experiments, we demonstrate that our principled initialisation effectively accelerates learning in ICNNs and leads to better generalisation. Moreover, we find that, in contrast to common belief, ICNNs can be trained without skip-connections when initialised correctly. Finally, we apply ICNNs to a real-world drug discovery task and show that they allow for more effective molecular latent space exploration.
Optimal transport mapping via input convex neural networks
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Irvine(加州大学欧文分校);Princeton University(普林斯顿大学);University of Washington(华盛顿大学)
A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimization
College of Computer Science and Technology(计算机科学与技术学院);College of Computer and Data Science(计算机与数据科学学院);College of Information Engineering(信息工程学院);School of Science(科学学院);Shandong Provincial Key Laboratory of Network Based Intelligent Computing(网络基于智能计算的山东省重点实验室)
Archetypal Graph Generative Models: Explainable and Identifiable Communities via Anchor-Dominant Convex Hulls
Yale Institute for Network Science, Yale University, USA(耶鲁网络科学研究所,耶鲁大学,美国);Université Paris Saclay, Université Paris Cité, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli, France(巴黎萨克雷大学,巴黎城市大学,巴黎萨克雷高等学院,法国国家科学研究中心,SSA,INSERM,Borelli中心,法国);École Polytechnique, Paris, France(巴黎高等理工学院,巴黎,法国);Department of Informatics and Telecommunications, University of Peloponnese, Greece(信息与电信系,希腊皮埃利翁大学)
Weakly Convex Ridge Regularization for 3D Non-Cartesian MRI Reconstruction
Faculty of Mathematics, Chemnitz University of Technology(开姆尼茨工业大学数学学院);NeuroSpin, Frédéric Joliot Institute for Life Sciences, CEA Paris-Saclay(法国原子能委员会巴黎-萨克雷大学弗雷德里克·约里奥生命科学研究所神经自旋中心);MIND, Inria Saclay Centre(法国国家信息与自动化研究所萨克雷中心MIND团队);Siemens Healthineers(西门子医疗)
Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks
Institute of Mathematics, Faculty of Mechanical Engineering, Brno University of Technology(布拉格技术大学数学系,机械工程系);Université de Toulouse, INSA Toulouse, CNRS, IMT(图卢兹大学,图卢兹INSA,CNRS,IMT);University of Warsaw(华沙大学)
Provable Responsible AI and Data Analytics (PRADA) Lab(可证明责任AI和数据分析实验室);King Abdullah University of Science and Technology(卡瓦尔大学科学与技术大学);State University of New York at Buffalo(纽约州立大学布法罗分校);Technische Universität Berlin(柏林技术大学)
Solving Max-Cut to Global Optimality via Feasibility-Preserving Graph Neural Networks
Davidson School of Chemical Engineering, Purdue University(普渡大学化学工程大卫逊学院);Faculty of Computer Science, RWTH Aachen University(亚琛工业大学计算机科学学院);Jacobs Technion-Cornell Institute, Cornell Tech(技术学院-康奈尔研究所)