arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1707.08308cs.LG
收藏

张量回归网络

Tensor Regression Networks

NVIDIA(英伟达);Imperial College London(帝国理工学院);Carnegie Mellon University(卡内基梅隆大学);Amazon AI(亚马逊人工智能);University of Southern California(南加州大学);California Institute of Technology(加州理工学院)

Jean Kossaifi, Zachary C. Lipton, Arinbjorn Kolbeinsson, Aran Khanna, Tommaso Furlanello, Anima Anandkumar

摘要:

卷积神经网络通常由许多卷积层和一个或多个全连接层组成。虽然卷积层在高阶激活张量之间映射,但完全连接的层对扁平化的激活向量进行操作。尽管在经验上取得了成功,但这种方法有明显的缺点。展平后的全连接层放弃了激活中的多线性结构,需要许多参数。我们通过结合张量代数运算来解决这些问题,这些运算在每一层都保留了多线性结构。首先,我们介绍了张量收缩层(Tensor Contraction Layers,TCL),它可以降低输入的维度,同时使用张量收缩保持其多线性结构。接下来,我们介绍张量回归层(TRL),它通过从高阶激活张量到任意阶输出张量的低秩多线性映射来表达输出。我们端到端地学习收缩和回归因子,并使用更少的参数生成准确的网络。此外,我们的层通过对激活(TCL)和回归权重(TRL)施加低秩约束来正则化网络。ImageNet上的实验表明,应用于VGG和ResNet架构,与完全连接的层相比,TCL和TRL减少了65%以上的参数数量,同时保持或提高了准确性。除了节省空间外,我们的方法利用拓扑结构的能力对于MRI等结构化数据至关重要。特别是,我们展示了与英国生物银行数据集相关的三个任务的可比架构的显着性能改进。

英文摘要:

Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high-order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs), which express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end-to-end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low-rank constraints on the activations (TCL) and regression weights (TRL). Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy. In addition to the space savings, our approach's ability to leverage topological structure can be crucial for structured data such as MRI. In particular, we demonstrate significant performance improvements over comparable architectures on three tasks associated with the UK Biobank dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑