神经网络的容量分配分析:原则性体系结构设计的工具
Capacity allocation analysis of neural networks: A tool for principled architecture design
摘要:
设计神经网络架构是一项介于科学和艺术之间的任务。对于给定的任务,某些架构最终会比其他架构更受欢迎,这是基于直觉、经验、实验和运气的混合。对于许多任务来说,最终的结果归因于损失函数,而对于其他一些任务来说,需要进一步的感知评估来评估和比较模型的性能。在本文中,我们介绍了容量分配分析的概念,目的是阐明什么网络架构集中他们的建模能力,当用于一个给定的任务。我们更特别地关注空间容量分配,它分析了后验的有效数量的参数,一个给定的模型已分配用于建模依赖于一个给定的点或区域在输入空间中,在线性设置。我们使用这个框架来执行一些经典的架构之间的定量比较各种合成任务。最后,我们考虑如何在非线性设置的容量分配可能会翻译。
英文摘要:
Designing neural network architectures is a task that lies somewhere between science and art. For a given task, some architectures are eventually preferred over others, based on a mix of intuition, experience, experimentation and luck. For many tasks, the final word is attributed to the loss function, while for some others a further perceptual evaluation is necessary to assess and compare performance across models. In this paper, we introduce the concept of capacity allocation analysis, with the aim of shedding some light on what network architectures focus their modelling capacity on, when used on a given task. We focus more particularly on spatial capacity allocation, which analyzes a posteriori the effective number of parameters that a given model has allocated for modelling dependencies on a given point or region in the input space, in linear settings. We use this framework to perform a quantitative comparison between some classical architectures on various synthetic tasks. Finally, we consider how capacity allocation might translate in non-linear settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。