arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1808.01405cs.CV
收藏

教师指导的架构搜索

Teacher Guided Architecture Search

McGovern Institute for Brain Research(麦戈文脑研究所);MIT(麻省理工学院);University of Amsterdam(阿姆斯特丹大学)

Pouya Bashivan, Mark Tensen, James J DiCarlo

摘要:

最近计算机视觉的神经网络的大部分改进是由于发现了新的网络架构。大多数先前的工作都使用了候选模型在有限训练后的性能,以可行的方式自动引导搜索。通过测量具有未知详细结构的高性能网络(例如灵长类动物视觉系统)来引导搜索,是否可以进一步提高计算效率?作为实现这一目标的一步,我们使用表征相似性分析来评估候选网络与(固定的,高性能的)教师网络的内部激活的相似性。我们表明,采用这种评估指标可以产生一个数量级的搜索效率超过性能导向的方法。我们的方法找到了一种卷积细胞结构,其性能与以前使用其他方法发现的相似,但总计算成本比神经架构搜索(NAS)低两个数量级,比渐进式神经架构搜索(PNAS)低四倍以上。我们进一步表明,来自灵长类视觉系统的仅约300个神经元的测量提供了足够的信号来找到具有Imagenet top-1错误的网络,该错误显着低于单独通过性能引导的架构搜索实现的错误。这些结果表明,代表性匹配可以用来加速网络架构搜索的情况下,其中一个有机会获得一些或所有的内部表示的教师网络的兴趣,如大脑的感觉处理网络。

英文摘要:

Much of the recent improvement in neural networks for computer vision has resulted from discovery of new networks architectures. Most prior work has used the performance of candidate models following limited training to automatically guide the search in a feasible way. Could further gains in computational efficiency be achieved by guiding the search via measurements of a high performing network with unknown detailed architecture (e.g. the primate visual system)? As one step toward this goal, we use representational similarity analysis to evaluate the similarity of internal activations of candidate networks with those of a (fixed, high performing) teacher network. We show that adopting this evaluation metric could produce up to an order of magnitude in search efficiency over performance-guided methods. Our approach finds a convolutional cell structure with similar performance as was previously found using other methods but at a total computational cost that is two orders of magnitude lower than Neural Architecture Search (NAS) and more than four times lower than progressive neural architecture search (PNAS). We further show that measurements from only ~300 neurons from primate visual system provides enough signal to find a network with an Imagenet top-1 error that is significantly lower than that achieved by performance-guided architecture search alone. These results suggest that representational matching can be used to accelerate network architecture search in cases where one has access to some or all of the internal representations of a teacher network of interest, such as the brain's sensory processing networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑