Differentiation lies at the core of many machine-learning algorithms, and is well-supported by popular autodiff systems, such as TensorFlow and PyTorch. Originally, these systems have been developed to compute derivatives of differentiable functions, but in practice, they are commonly applied to functions with non-differentiabilities. For instance, neural networks using ReLU define non-differentiable functions in general, but the gradients of losses involving those functions are computed using autodiff systems in practice. This status quo raises a natural question: are autodiff systems correct in any formal sense when they are applied to such non-differentiable functions? In this paper, we provide a positive answer to this question. Using counterexamples, we first point out flaws in often-used informal arguments, such as: non-differentiabilities arising in deep learning do not cause any issues because they form a measure-zero set. We then investigate a class of functions, called PAP functions, that includes nearly all (possibly non-differentiable) functions in deep learning nowadays. For these PAP functions, we propose a new type of derivatives, called intensional derivatives, and prove that these derivatives always exist and coincide with standard derivatives for almost all inputs. We also show that these intensional derivatives are what most autodiff systems compute or try to compute essentially. In this way, we formally establish the correctness of autodiff systems applied to non-differentiable functions.
DQC: a Python program package for Differentiable Quantum Chemistry
University of Oxford(牛津大学);Molecular Sciences Software Institute(分子科学软件研究所);STFC Rutherford Appleton Laboratory(STFC卢瑟福·阿普尔顿实验室)
自动微分代表了科学编程中的范式转变,大多数应用程序都需要评估函数及其导数。通过消除显式导出梯度表达式的需要,可以缩短开发时间,并简化计算。由于这些原因,自动微分在过去十年中推动了各种复杂机器学习技术的快速发展,但现在也越来越多地显示出其支持量子系统的{ \it ab initio}模拟和增强计算量子化学的价值。在这里,我们提出了一个开源的可微量子化学模拟代码,DQC,并探索自动微分促进的应用:(1)计算分子微扰特性;(2)重新优化碳氢化合物的基组;(3)检查自洽场波函数的稳定性;(4)通过炼金术微扰预测分子特性。
Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation
Khoury College of Computer Sciences, Northeastern University(东北大学Khoury计算机科学学院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Shanghai Qi Zhi Institute(上海期智研究院)
Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
Toulouse School of Economics(图卢兹经济学院);Université Toulouse 1 Capitole(图卢兹第一大学);IRIT(图卢兹计算机研究所);Université de Toulouse(图卢兹大学);IRT Saint Exupery(圣埃克苏佩里技术研究院)
Differentiable Quantum Programming with Unbounded Loops
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);University of Maryland(马里兰大学)
VertAX: a differentiable vertex model for learning epithelial tissue mechanics
Center for Interdisciplinary Research in Biology (CIRB), Collège de France, Université PSL, CNRS, INSERM(跨学科生物学研究中心(CIRB),法兰西公学院,巴黎文理研究大学,法国国家科学研究中心,法国国家健康与医学研究院);Center for Integrative Biology (CBI), University of Toulouse, CNRS(整合生物学中心(CBI),图卢兹大学,法国国家科学研究中心);Center for Life Nano & Neuro Science, IIT(生命纳米与神经科学中心,意大利技术研究院);Institute of Heritage Science, CNR(遗产科学研究所,意大利国家研究委员会);Department of Physics, Università Sapienza(物理系,罗马大学);Google DeepMind(谷歌DeepMind)
Differentiable Expectation-Maximisation and Applications to Gaussian Mixture Model Optimal Transport
Université Paris Cité, CNRS, MAP5(巴黎大学、国家科学研究中心、MAP5);CNRS(国家科学研究中心);ENS Paris-Saclay(巴黎-萨克雷大学);ECOLE POLYTECHNIQUE(巴黎高等理工学院);Institut Polytechnique de Paris(巴黎理工学院)
Differentiate the Solver, Not the Equation: Reverse-Sweep Adjoints for Block Implicit Simulation
Stanford University(斯坦福大学);University of California, Los Angeles(加州大学洛杉矶分校);LightSpeed Studios, Tencent America(腾讯美国光速工作室);The University of Utah(犹他大学)
Pymanopt: A Python Toolbox for Optimization on Manifolds using Automatic Differentiation
University College London(伦敦大学学院);RWTH Aachen University(亚琛工业大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)
流形上的优化(英语:Optimization on manifold)是一类优化目标函数的方法,在某种意义上说,满足约束的点的集合允许可微流形的结构。虽然许多优化问题的描述的形式,微分几何的技术性和繁琐的计算导数构成了一个显着的障碍,用这些方法进行实验。我们介绍Pymanopt(可在https://pymanopt.github.io上获得),这是一个用Python实现的流形优化工具箱,与Manopt Matlab工具箱类似,它实现了几种流形几何和优化算法。此外,我们通过使用自动微分来计算衍生信息,进一步降低了用户的障碍,节省了用户的时间,并使他们免受潜在的计算和实施错误的影响。
4D-Var using Hessian approximation and backpropagation applied to automatically-differentiable numerical and machine learning models
University of Colorado Boulder(科罗拉多大学博尔德分校);Sofar Ocean(索法海洋公司);Cooperative Institute for Research in Environmental Sciences at the University of Colorado Boulder(科罗拉多大学博尔德分校合作环境科学研究所);Google Research(谷歌研究院)
Automatic differentiation of nonsmooth iterative algorithms
Toulouse School of Economics, University of Toulouse Capitole(图卢兹经济学院,图卢兹第一大学);IRIT, CNRS, Université de Toulouse, ANITI(图卢兹信息学研究所,法国国家科学研究中心,图卢兹大学,图卢兹人工智能与自然智能研究所)
Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems
Toulouse School of Economics(图卢兹经济学院);University of Toulouse(图卢兹大学);IRIT(图卢兹计算机科学研究所);CNRS(法国国家科学研究中心);Université Toulouse III Paul Sabatier(图卢兹第三大学保罗·萨巴蒂埃大学);Institut Universitaire de France(法兰西大学研究院);CVN(认知与神经计算研究中心);CentraleSupélec(巴黎中央理工-高等电力学院);Université Paris-Saclay(巴黎萨克雷大学)
Learning Constrained Adaptive Differentiable Predictive Control Policies With Guarantees
Pacific Northwest National Laboratory(太平洋西北国家实验室)
我们提出了可微预测控制(DPC),学习约束的线性系统的概率性能保证的神经控制策略的方法。我们采用自动微分获得直接的政策梯度反向传播的模型预测控制(MPC)的损失函数和约束的惩罚,通过可微的闭环系统动力学模型。我们证明了所提出的方法可以学习参数约束控制策略来稳定具有不稳定动态的系统,跟踪时变参考,并满足非线性状态和输入约束。与基于模仿学习的方法相比,我们的方法不依赖于监督控制器。最重要的是,我们证明,在不损失性能的情况下,我们的方法是可扩展的,计算效率比隐式,显式和近似MPC。IEEE Transactions on Automatic Control(IEEE自动控制学报)
Learning vertical coordinates via automatic differentiation of a dynamical core
Centre pour l’étude et la simulation du climat à l’échelle régionale (ESCER), Département des Sciences de la Terre et de l’Atmosphère, Université du Québec à Montréal(区域气候研究与模拟中心(ESCER)、地球科学与大气科学系、魁北克大学蒙特利尔分校);PIMS, Department of Computer Science, University of Saskatchewan(PIMS、计算机科学系、萨斯喀彻温大学);Department of Mathematics and Statistics, Memorial University of Newfoundland(数学与统计学系、纽芬兰纪念大学)