Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field. Specifically, representations of the inputs are meta-learned such that a task-specific linear classifier is obtained as a solution of an ordinary differential equation (ODE). Treating the learning process as an ODE offers the notable advantage that the length of the trajectory is now continuous, as opposed to a fixed and discrete number of gradient steps. As a consequence, we can optimize the amount of adaptation necessary to solve a new task using stochastic gradient descent, in addition to learning the initial conditions as is standard practice in gradient-based meta-learning. Importantly, in order to compute the exact meta-gradients required for the outer-loop updates, we devise an efficient algorithm based on forward mode differentiation, whose memory requirements do not scale with the length of the learning trajectory, thus allowing longer adaptation in constant memory. We provide analytical guarantees for the stability of COMLN, we show empirically its efficiency in terms of runtime and memory usage, and we illustrate its effectiveness on a range of few-shot image classification problems.
Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training
Shanghai Jiao Tong University（上海交通大学）；Shanghai Qi Zhi Institute（上海期智研究院）；Alibaba Group（阿里巴巴集团）；Microsoft Research（微软研究院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））
Randomized Forward Mode of Automatic Differentiation For Optimization Algorithms
Division of Applied Mathematics, Brown University（布朗大学应用数学系）；Department of Mathematics, North Carolina State University（北卡罗来纳州立大学数学系）；Mathematical Institute for Data Science, Pohang University of Science and Technology (POSTECH)（浦项科技大学数据科学数学研究所）
Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation
Khoury College of Computer Sciences, Northeastern University（东北大学Khoury计算机科学学院）；Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；Shanghai Qi Zhi Institute（上海期智研究院）
Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State
Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Center for Data Science, Academy for Advanced Interdisciplinary Studies, Peking University（北京大学前沿交叉学科研究院数据科学研究中心）；Pazhou Lab（琶洲实验室）
An automatic-differentiation framework for time-lapse electrical resistivity tomography inversion of hydrologic dynamics
School of Geosciences and Info-physics, Central South University（中南大学地球科学与信息物理学院）；School of Earth, Environment, and Sustainability, University of Iowa（爱荷华大学地球、环境与可持续发展学院）；School of Management Science and Engineering, Hunan University of Technology and Business（湖南工商大学管理科学与工程学院）
On the complexity of nonsmooth automatic differentiation
Toulouse School of Economics（图卢兹经济学院）；Université de Toulouse（图卢兹大学）；Thales LAS France（法国泰勒斯激光系统公司）；IRIT, CNRS（法国国家科学研究中心信息与电信研究所）；Institut Universitaire de France (IUF)（法国大学研究院）
Local learning for stable backpropagation-free neural network training towards physical learning
Department of Materials Science and Engineering, Delft University of Technology（材料科学与工程系，代尔夫特理工大学）；Department of Precision and Microsystems Engineering, Delft University of Technology（精密与微系统工程系，代尔夫特理工大学）；Department of Intelligent Systems, Delft University of Technology（智能系统系，代尔夫特理工大学）
Automatic differentiation of nonsmooth iterative algorithms
Toulouse School of Economics, University of Toulouse Capitole（图卢兹经济学院，图卢兹第一大学）；IRIT, CNRS, Université de Toulouse, ANITI（图卢兹信息学研究所，法国国家科学研究中心，图卢兹大学，图卢兹人工智能与自然智能研究所）
Differentiate the Solver, Not the Equation: Reverse-Sweep Adjoints for Block Implicit Simulation
Stanford University（斯坦福大学）；University of California, Los Angeles（加州大学洛杉矶分校）；LightSpeed Studios, Tencent America（腾讯美国光速工作室）；The University of Utah（犹他大学）
VertAX: a differentiable vertex model for learning epithelial tissue mechanics
Center for Interdisciplinary Research in Biology (CIRB), Collège de France, Université PSL, CNRS, INSERM（跨学科生物学研究中心（CIRB），法兰西公学院，巴黎文理研究大学，法国国家科学研究中心，法国国家健康与医学研究院）；Center for Integrative Biology (CBI), University of Toulouse, CNRS（整合生物学中心（CBI），图卢兹大学，法国国家科学研究中心）；Center for Life Nano & Neuro Science, IIT（生命纳米与神经科学中心，意大利技术研究院）；Institute of Heritage Science, CNR（遗产科学研究所，意大利国家研究委员会）；Department of Physics, Università Sapienza（物理系，罗马大学）；Google DeepMind（谷歌DeepMind）
