arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.00600cs.CL
收藏

最小描述长度循环神经网络

Minimum Description Length Recurrent Neural Networks

Ecole Normale Supérieure(巴黎高等师范学院);Tel Aviv University(特拉维夫大学);EHESS(法国高等社会科学研究院);PSL University(巴黎文理研究大学);CNRS(法国国家科学研究中心)

Nur Lan, Michal Geyer, Emmanuel Chemla, Roni Katzir

摘要:

我们训练神经网络以优化最小描述长度分数,即,以平衡网络的复杂性和任务的准确性。我们表明,优化这个目标函数的网络掌握了涉及记忆挑战的任务,并超越了上下文无关的语言。这些学习者掌握$a^nb^n$、$a^nb^nc^n$、$a^nb^{2n}$、$a^nb^mc^{n+m}$等语言,并进行加法运算。而且,他们经常以100%的准确度这样做。这些网络很小,其内部运作是透明的。因此,我们提供了正式的证明,他们的完美的准确性,不仅在一个给定的测试集,但对于任何输入序列。据我们所知,没有其他的连接主义模型已经被证明可以完全概括地捕捉这些语言的底层语法。

英文摘要:

We train neural networks to optimize a Minimum Description Length score, i.e., to balance between the complexity of the network and its accuracy at a task. We show that networks optimizing this objective function master tasks involving memory challenges and go beyond context-free languages. These learners master languages such as $a^nb^n$, $a^nb^nc^n$, $a^nb^{2n}$, $a^nb^mc^{n+m}$, and they perform addition. Moreover, they often do so with 100% accuracy. The networks are small, and their inner workings are transparent. We thus provide formal proofs that their perfect accuracy holds not only on a given test set, but for any input sequence. To our knowledge, no other connectionist model has been shown to capture the underlying grammars for these languages in full generality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑