arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.12474cs.LGstat.ML
收藏

输入凸神经网络的原则性权重初始化

Principled Weight Initialisation for Input-Convex Neural Networks

LIT AI Lab(LIT人工智能实验室);ELLIS Unit Linz(林茨ELLIS单元);Institute for Machine Learning(机器学习研究所);Johannes Kepler University(约翰内斯·开普勒大学)

Pieter-Jan Hoedt, Günter Klambauer

AI总结:

本研究提出一种针对输入凸神经网络的原则性权重初始化方法,通过推广信号传播理论去除中心化假设,加速学习并提升泛化,且无需跳跃连接,在药物发现中实现更有效的分子空间探索。

摘要:

输入凸神经网络(ICNNs)是保证其输入输出映射具有凸性的网络。这些网络已成功应用于基于能量的建模、最优传输问题以及学习不变性。ICNNs的凸性通过使用非递减凸激活函数和非负权重来实现。由于这些特性,先前隐含假设权重中心化的初始化策略对ICNNs并不有效。通过研究非负权重层中的信号传播,我们能够推导出ICNNs的原则性权重初始化方法。具体而言,我们通过去除权重从中心化分布采样的假设,推广了信号传播理论。在一系列实验中,我们证明我们的原则性初始化能有效加速ICNNs的学习并带来更好的泛化性能。此外,我们发现与普遍看法相反,当正确初始化时,ICNNs可以在没有跳跃连接的情况下进行训练。最后,我们将ICNNs应用于一个现实世界的药物发现任务,并展示它们能够实现更有效的分子潜在空间探索。

英文摘要:

Input-Convex Neural Networks (ICNNs) are networks that guarantee convexity in their input-output mapping. These networks have been successfully applied for energy-based modelling, optimal transport problems and learning invariances. The convexity of ICNNs is achieved by using non-decreasing convex activation functions and non-negative weights. Because of these peculiarities, previous initialisation strategies, which implicitly assume centred weights, are not effective for ICNNs. By studying signal propagation through layers with non-negative weights, we are able to derive a principled weight initialisation for ICNNs. Concretely, we generalise signal propagation theory by removing the assumption that weights are sampled from a centred distribution. In a set of experiments, we demonstrate that our principled initialisation effectively accelerates learning in ICNNs and leads to better generalisation. Moreover, we find that, in contrast to common belief, ICNNs can be trained without skip-connections when initialised correctly. Finally, we apply ICNNs to a real-world drug discovery task and show that they allow for more effective molecular latent space exploration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑