用最小描述长度弥合神经网络形式语言学习中的经验与理论差距
Bridging the Empirical-Theoretical Gap in Neural Network Formal Language Learning Using Minimum Description Length
AI总结:
本文通过一种简单形式语言学习任务证明理论正确解并非常用目标的最优解,并提出以最小描述长度目标替代标准目标可使正确解成为最优解。
摘要:
神经网络在许多任务上提供了良好的近似能力,但即使理论工作表明某些架构可以表达完美解,它们也始终无法达到完美泛化。以形式语言学习任务为例,我们聚焦于一种简单的形式语言,并证明理论上正确的解实际上并不是常用目标函数的最优解——即便使用了根据常识应当导向简单权重和良好泛化的正则化技术(L1、L2)或其他元启发式方法(早停、dropout)也是如此。另一方面,将标准目标替换为最小描述长度目标(MDL)后,正确的解成为最优解。
英文摘要:
Neural networks offer good approximation to many tasks but consistently fail to reach perfect generalization, even when theoretical work shows that such perfect solutions can be expressed by certain architectures. Using the task of formal language learning, we focus on one simple formal language and show that the theoretically correct solution is in fact not an optimum of commonly used objectives -- even with regularization techniques that according to common wisdom should lead to simple weights and good generalization (L1, L2) or other meta-heuristics (early-stopping, dropout). On the other hand, replacing standard targets with the Minimum Description Length objective (MDL) results in the correct solution being an optimum.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。