Derivatives of differential equation solutions are commonly for parameter estimation, fitting neural differential equations, and as model diagnostics. However, with a litany of choices and a Cartesian product of potential methods, it can be difficult for practitioners to understand which method is likely to be the most effective on their particular application. In this manuscript we investigate the performance characteristics of Discrete Local Sensitivity Analysis implemented via Automatic Differentiation (DSAAD) against continuous adjoint sensitivity analysis. Non-stiff and stiff biological and pharmacometric models, including a PDE discretization, are used to quantify the performance of sensitivity analysis methods. Our benchmarks show that on small systems of ODEs (approximately $<100$ parameters+ODEs), forward-mode DSAAD is more efficient than both reverse-mode and continuous forward/adjoint sensitivity analysis. The scalability of continuous adjoint methods is shown to be more efficient than discrete adjoints and forward methods after crossing this size range. These comparative studies demonstrate a trade-off between memory usage and performance in the continuous adjoint methods that should be considered when choosing the technique, while numerically unstable backsolve techniques from the machine learning literature are demonstrated as unsuitable for most scientific models. The performance of adjoint methods is shown to be heavily tied to the reverse-mode AD method, with tape-based AD methods shown to be 2 orders of magnitude slower on nonlinear partial differential equations than static AD techniques. These results also demonstrate the applicability of DSAAD to differential-algebraic equations, delay differential equations, and hybrid differential equation systems, showcasing an ease of implementation advantage for DSAAD approaches.
LVMT, ENPC, Institut Polytechnique de Paris, Univ Gustave Eiffel(巴黎高等路桥学院,巴黎理工学院,古斯塔夫·埃菲尔大学,LVMT实验室);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所(BIFOLD));Machine Learning Group, Technical University of Berlin(柏林工业大学机器学习组)
AuTO: A Framework for Automatic differentiation in Topology Optimization
拓扑优化的一个关键步骤是求灵敏度。灵敏度的手动推导和实现可能非常费力且容易出错,特别是对于重要的目标、约束和材料模型。另一种方法是利用自动微分(AD)。虽然AD已经存在了几十年,并且也被应用于TO,但在很大程度上没有更广泛的采用。在这篇教育论文中,我们的目标是重新介绍AD for TO,并通过说明性代码使其易于访问。特别是,我们采用了JAX,一个高性能的Python库,用于自动计算用户定义的TO问题的敏感性。由此产生的框架,这里称为AuTO,说明通过几个例子,在遵守最小化,符合机制设计和微结构设计。
Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds
Marchuk Institute of Numerical Mathematics of the Russian Academy of Sciences(俄罗斯科学院马尔丘克计算数学研究所);HSE University(高等经济学院);Skolkovo Institute of Science and Technology(斯科尔科沃科学技术学院)