arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2105.05031stat.MLcs.LGstat.APstat.CO
收藏

距离优化自适应步长的梯度流编码

Gradient flow encoding with distance optimization adaptive step size

ETH Zurich(苏黎世联邦理工学院)

Kyriakos Flouris, Anna Volokitin, Gustav Bredell, Ender Konukoglu

摘要:

自动编码器模型使用编码器将数据样本映射到较低维度的潜在空间,然后使用解码器将潜在空间表示映射回数据空间。隐含地,它依赖于编码器来近似解码器网络的逆,使得样本可以忠实地映射到潜在空间并从潜在空间返回。这种近似可能导致次优的潜在空间表示。在这项工作中,我们研究了一种仅解码器的方法,该方法使用梯度流对潜在空间中的数据样本进行编码。梯度流是基于给定的解码器定义的,旨在通过优化为任何给定的样本找到最佳的潜在空间表示,从而消除了通过编码器进行近似求逆的需要。通过常微分方程(ODE)实现梯度流,我们利用伴随方法来训练给定的解码器。我们进一步表明,伴随方法中的昂贵的积分经验可能不是完全必要的。此外,我们提出了一个$2^{nd}$阶常微分方程的方法,它近似Nesterov的加速梯度下降,每次迭代收敛速度更快。通常使用的ODE求解器可能对积分步长非常敏感,这取决于ODE的刚度。为了克服梯度流编码的敏感性,我们使用了一个自适应求解器,在每个积分步骤中优先考虑最小化损失。我们评估所提出的方法相比,自动编码模型。在我们的实验中,GFE表现出比自动编码模型更高的数据效率,这对于数据稀缺的应用程序至关重要。

英文摘要:

The autoencoder model uses an encoder to map data samples to a lower dimensional latent space and then a decoder to map the latent space representations back to the data space. Implicitly, it relies on the encoder to approximate the inverse of the decoder network, so that samples can be mapped to and back from the latent space faithfully. This approximation may lead to sub-optimal latent space representations. In this work, we investigate a decoder-only method that uses gradient flow to encode data samples in the latent space. The gradient flow is defined based on a given decoder and aims to find the optimal latent space representation for any given sample through optimisation, eliminating the need of an approximate inversion through an encoder. Implementing gradient flow through ordinary differential equations (ODE), we leverage the adjoint method to train a given decoder. We further show empirically that the costly integrals in the adjoint method may not be entirely necessary. Additionally, we propose a $2^{nd}$ order ODE variant to the method, which approximates Nesterov's accelerated gradient descent, with faster convergence per iteration. Commonly used ODE solvers can be quite sensitive to the integration step-size depending on the stiffness of the ODE. To overcome the sensitivity for gradient flow encoding, we use an adaptive solver that prioritises minimising loss at each integration step. We assess the proposed method in comparison to the autoencoding model. In our experiments, GFE showed a much higher data-efficiency than the autoencoding model, which can be crucial for data scarce applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑