arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.04205cs.LG
收藏

扩展 Forward Forward 算法

Extending the Forward Forward Algorithm

Saumya Gandhi, Ritu Gala, Jonah Kornberg, Advaith Sridhar

AI总结:

本文复现 Forward Forward 的 MNIST 实验,首次将其用于 IMDb 情感分析,并提出损失阈值的金字塔式优化策略,使测试误差最高改善 8%。

摘要:

Forward Forward 算法由 Geoffrey Hinton 于 2022 年 11 月提出,是一种训练神经网络的新方法,可作为反向传播的替代方案。在本项目中,我们复现了 Hinton 在 MNIST 数据集上的实验,随后通过两项重要贡献扩展了该方法的适用范围。首先,我们在 IMDb 电影评论数据集上建立了 Forward Forward 网络的基线性能。据我们所知,我们在这一情感分析任务上的结果标志着该算法首次被扩展到计算机视觉之外。其次,我们针对损失阈值这一 Forward Forward 方法特有的超参数,提出了一种新颖的金字塔式优化策略。我们的金字塔式方法表明,良好的阈值策略可使测试误差产生最高达 8% 的差异。最后,我们对训练后的参数进行了可视化,并得出若干重要洞见,例如 Forward Forward 网络所得权重的均值和方差明显更大(达到 10–20 倍)。代码仓库:https://github.com/Ads-cmu/ForwardForward

英文摘要:

The Forward Forward algorithm, proposed by Geoffrey Hinton in November 2022, is a novel method for training neural networks as an alternative to backpropagation. In this project, we replicate Hinton's experiments on the MNIST dataset, and subsequently extend the scope of the method with two significant contributions. First, we establish a baseline performance for the Forward Forward network on the IMDb movie reviews dataset. As far as we know, our results on this sentiment analysis task marks the first instance of the algorithm's extension beyond computer vision. Second, we introduce a novel pyramidal optimization strategy for the loss threshold - a hyperparameter specific to the Forward Forward method. Our pyramidal approach shows that a good thresholding strategy causes a difference of up to 8% in test error. Lastly, we perform visualizations of the trained parameters and derived several significant insights, such as a notably larger (10-20x) mean and variance in the weights acquired by the Forward Forward network. Repository: https://github.com/Ads-cmu/ForwardForward

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑