SoftJAX & SoftTorch: 通过信息性梯度增强自动微分库
SoftJAX & SoftTorch: Empowering Automatic Differentiation Libraries with Informative Gradients
AI总结:
SoftJAX与SoftTorch通过提供信息性梯度的软函数,增强了自动微分库的功能,使软松弛工具更易用。
摘要:
自动微分(AD)框架如JAX和PyTorch已使基于梯度的优化在许多科学领域得以应用。然而,这些库中的许多
英文摘要:
Automatic differentiation (AD) frameworks such as JAX and PyTorch have enabled gradient-based optimization for a wide range of scientific fields. Yet, many "hard" primitives in these libraries such as thresholding, Boolean logic, discrete indexing, and sorting operations yield zero or undefined gradients that are not useful for optimization. While numerous "soft" relaxations have been proposed that provide informative gradients, the respective implementations are fragmented across projects, making them difficult to combine and compare. This work introduces SoftJAX and SoftTorch, open-source, feature-complete libraries for soft differentiable programming. These libraries provide a variety of soft functions as drop-in replacements for their hard JAX and PyTorch counterparts. This includes (i) elementwise operators such as clip or abs, (ii) utility methods for manipulating Booleans and indices via fuzzy logic, (iii) axiswise operators such as sort or rank -- based on optimal transport or permutahedron projections, and (iv) offer full support for straight-through gradient estimation. Overall, SoftJAX and SoftTorch make the toolbox of soft relaxations easily accessible to differentiable programming, as demonstrated through benchmarking and a practical case study. Code is available at github.com/a-paulus/softjax and github.com/a-paulus/softtorch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。