First order methods, which solely rely on gradient information, are commonly used in diverse machine learning (ML) and data analysis (DA) applications. This is attributed to the simplicity of their implementations, as well as low per-iteration computational/storage costs. However, they suffer from significant disadvantages; most notably, their performance degrades with increasing problem ill-conditioning. Furthermore, they often involve a large number of hyper-parameters, and are notoriously sensitive to parameters such as the step-size. By incorporating additional information from the Hessian, second-order methods, have been shown to be resilient to many such adversarial effects. However, these advantages of using curvature information come at the cost of higher per-iteration costs, which in \enquote{big data} regimes, can be computationally prohibitive.
In this paper, we show that, contrary to conventional belief, second-order methods, when implemented appropriately, can be more efficient than first-order alternatives in many large-scale ML/ DA applications. In particular, in convex settings, we consider variants of classical Newton\textsf{'}s method in which the Hessian and/or the gradient are randomly sub-sampled. We show that by effectively leveraging the power of GPUs, such randomized Newton-type algorithms can be significantly accelerated, and can easily outperform state of the art implementations of existing techniques in popular ML/ DA software packages such as TensorFlow. Additionally these randomized methods incur a small memory overhead compared to first-order methods. In particular, we show that for million-dimensional problems, our GPU accelerated sub-sampled Newton\textsf{'}s method achieves a higher test accuracy in milliseconds as compared with tens of seconds for first order alternatives.
Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample
University of Michigan(密歇根大学);Lehigh University(利哈伊大学);KAUST(阿卜杜拉国王科技大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学)
Unpaired Deep Learning for Accelerated MRI using Optimal Transport Driven CycleGAN
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST));Seoul National University College of Medicine, Seoul National University Bundang Hospital(首尔大学医学院,首尔大学盆唐医院)
Deep Learning-based Accelerated MR Cholangiopancreatography without Fully-sampled Data
Friedrich-Alexander-Universität Erlangen-Nürnberg(弗里德里希-亚历山大-埃尔朗根-纽伦堡大学);Siemens Healthineers(西门子医疗);New York University Grossman School of Medicine(纽约大学格罗斯曼医学院)
Physics-informed self-supervised deep learning reconstruction for accelerated first-pass perfusion cardiac MRI
Universidad de Valladolid(巴利亚多利德大学);King’s College London(伦敦国王学院);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);Centre for Marine Sciences - CCMAR(海洋科学中心(CCMAR))
Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
Stanford University(斯坦福大学);University of Queensland(昆士兰大学);International Computer Science Institute(国际计算机科学研究所);University of California at Berkeley(加州大学伯克利分校)