arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.06179cs.LGstat.ML
收藏

超现实词典学习中的恢复与泛化

Recovery and Generalization in Over-Realized Dictionary Learning

Johns Hopkins University(约翰斯·霍普金斯大学);University of California, Berkeley(加州大学伯克利分校);Denver University(丹佛大学)

Jeremias Sulam, Chong You, Zhihui Zhu

摘要:

在二十多年的研究中,字典学习领域已经收集了大量成功的应用程序,只有在与底层字典相同的模型类中进行优化时,才知道模型恢复的理论保证。这项工作的特点是令人惊讶的现象,字典恢复可以通过搜索更大的过度实现模型的空间来促进。这种观察是普遍的,并且与所使用的特定字典学习算法无关。我们在实践中彻底证明了这一点,并通过将恢复措施与泛化界限联系起来来分析这一现象。特别是,我们表明,模型恢复可以由经验风险,模型依赖量和泛化差距的上限,反映了我们的实证研究结果。我们进一步表明,一个有效的和可证明正确的蒸馏方法可以用来恢复正确的原子从过度实现的模型。因此,我们的元算法提供了字典估计,并始终更好地恢复地面实况模型。

英文摘要:

In over two decades of research, the field of dictionary learning has gathered a large collection of successful applications, and theoretical guarantees for model recovery are known only whenever optimization is carried out in the same model class as that of the underlying dictionary. This work characterizes the surprising phenomenon that dictionary recovery can be facilitated by searching over the space of larger over-realized models. This observation is general and independent of the specific dictionary learning algorithm used. We thoroughly demonstrate this observation in practice and provide an analysis of this phenomenon by tying recovery measures to generalization bounds. In particular, we show that model recovery can be upper-bounded by the empirical risk, a model-dependent quantity and the generalization gap, reflecting our empirical findings. We further show that an efficient and provably correct distillation approach can be employed to recover the correct atoms from the over-realized model. As a result, our meta-algorithm provides dictionary estimates with consistently better recovery of the ground-truth model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑