When solving inverse problems, one increasingly popular approach is to use pre-trained diffusion models as plug-and-play priors. This framework can accommodate different forward models without re-training while preserving the generative capability of diffusion models. Despite their success in many imaging inverse problems, most existing methods rely on privileged information such as derivative, pseudo-inverse, or full knowledge about the forward model. This reliance poses a substantial limitation that restricts their use in a wide range of problems where such information is unavailable, such as in many scientific applications. We propose Ensemble Kalman Diffusion Guidance (EnKG), a derivative-free approach that can solve inverse problems by only accessing forward model evaluations and a pre-trained diffusion model prior. We study the empirical effectiveness of EnKG across various inverse problems, including scientific settings such as inferring fluid flows and astronomical objects, which are highly non-linear inverse problems that often only permit black-box access to the forward model. We open-source our code at https://github.com/devzhk/enkg-pytorch.
The standard probabilistic perspective on machine learning gives rise to empirical risk-minimization tasks that are frequently solved by stochastic gradient descent (SGD) and variants thereof. We present a formulation of these tasks as classical inverse or filtering problems and, furthermore, we propose an efficient, gradient-free algorithm for finding a solution to these problems using ensemble Kalman inversion (EKI). Applications of our approach include offline and online supervised learning with deep neural networks, as well as graph-based semi-supervised learning. The essence of the EKI pr…
Simple Approximation and Derivative Free Inference-Time Scaling for Diffusion Models via Sequential Monte Carlo on Path Measures
School of Mathematical Sciences, Peking University, Beijing, China；School of Mathematical Sciences, Fudan University, Shanghai, China；Department of Industrial Engineering \& Management Sciences, Northwestern University, Evanston, IL, United States；Institute for Computational \& Mathematical Engineering, Stanford University, Stanford, CA, United States；Management Science \& Engineering, Stanford University, Stanford, CA, United States
We study the problem of diffeomorphometric geodesic landmark matching where the objective is to find a diffeomorphism that via its group action maps between two sets of landmarks. It is well-known that the motion of the landmarks, and thereby the diffeomorphism, can be encoded by an initial momentum leading to a formulation where the landmark matching problem can be solved as an optimisation problem over such momenta. The novelty of our work lies in the application of a derivative-free Bayesian inverse method for learning the optimal momentum encoding the diffeomorphic mapping between the temp…
Flow Matching for Efficient and Scalable Data Assimilation
The Computing and Mathematical Sciences Department, California Institute of Technology（加州理工学院计算与数学科学系）；Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah（犹他大学数学与科学计算系和成像研究所）
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（前沿人工智能研究中心，科技研究局，新加坡）；Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，科技研究局，新加坡）；School of Mechanical Sciences, Indian Institute of Technology, Goa, India（机械科学学院，印度理工学院，果阿，印度）；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）
We introduce a deep neural network based method for solving a class of elliptic partial differential equations. We approximate the solution of the PDE with a deep neural network which is trained under the guidance of a probabilistic representation of the PDE in the spirit of the Feynman-Kac formula. The solution is given by an expectation of a martingale process driven by a Brownian motion. As Brownian walkers explore the domain, the deep neural network is iteratively trained using a form of reinforcement learning. Our method is a 'Derivative-Free Loss Method' since it does not require the exp…
The University of British Columbia（英属哥伦比亚大学）；Xtract Inc.（Xtract公司）；Centrum Wiskunde & Informatica (CWI)（数学与计算机科学中心）；University College London（伦敦大学学院）；Emory University（埃默里大学）
In this work, we present a new derivative-free optimization method and investigate its use for training neural networks. Our method is motivated by the Ensemble Kalman Filter (EnKF), which has been used successfully for solving optimization problems that involve large-scale, highly nonlinear dynamical systems. A key benefit of the EnKF method is that it requires only the evaluation of the forward propagation but not its derivatives. Hence, in the context of neural networks, it alleviates the need for back propagation and reduces the memory consumption dramatically. However, the method is not a…
PILOT: A Data-Free Continual Learning Approach for Real-Time Semantic Segmentation via Boundary Guidance
Department of Mathematics, College of Arts and Sciences, Embry-Riddle Aeronautical University（数学系，文理学院，埃姆布里-里德航空大学）；Department of Electrical Engineering and Computer Science, College of Engineering, Embry-Riddle Aeronautical University（电气工程与计算机科学系，工程学院，埃姆布里-里德航空大学）
