arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01976cs.LG
收藏

SPARLING:通过极稀疏激活学习潜在表示

SPARLING: Learning Latent Representations with Extremely Sparse Activations

Department of Electrical Engineering and Computer Science(电气工程与计算机科学系);Massachusetts Institute of Technology(麻省理工学院);Department of Computer and Information Science(计算机与信息科学系);University of Pennsylvania(宾夕法尼亚大学)

Kavi Gupta, Osbert Bastani, Armando Solar-Lezama

AI总结:

SPARLING通过极稀疏激活学习潜在表示,利用信息瓶颈技术实现高效中间状态建模,实验证明极端稀疏性对建模效果至关重要。

摘要:

现实世界的过程往往包含可以建模为极稀疏激活张量的中间状态。在本工作中,我们分析了这种稀疏和局部的潜在中间变量的可识别性,我们称之为motif。我们证明了我们的Motif可识别性定理,指出在某些假设下,可以通过减少端到端误差来精确识别这些motif。值得注意的是,我们不假设参数的可识别性,而是假设由局部模型输出的潜在中间表示,因此允许这些表示可以是输入的任意复杂函数。此外,我们提供了Sparling算法,该算法使用了一种新的信息瓶颈,强制实施其他技术无法达到的激活稀疏性水平。我们实证地证明了极端稀疏性对于获得良好的中间状态建模是必要的。在合成领域中,我们能够精确地将中间状态局部化到特征排列,准确率超过90%,即使只训练端到端。

英文摘要:

Real-world processes often contain intermediate state that can be modeled as an extremely sparse activation tensor. In this work, we analyze the identifiability of such sparse and local latent intermediate variables, which we call motifs. We prove our Motif Identifiability Theorem, stating that under certain assumptions it is possible to precisely identify these motifs exclusively by reducing end-to-end error. Notably, we do not assume identifiability of parameters, but rather of a latent intermediate representation output by a local model, thus allowing these representations to be arbitrarily complex functions of the input. Additionally, we provide the Sparling algorithm, which uses a new kind of informational bottleneck that enforces levels of activation sparsity unachievable using other techniques. We confirm empirically that extreme sparsity is necessary to achieve good intermediate state modeling. On synthetic domains, we are able to precisely localize the intermediate states up to feature permutation with > 90% accuracy, even though we only train end-to-end.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑