用并行级联网络和时间差损失改进任意时间预测
Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss
摘要：
虽然深度前馈神经网络与灵长类视觉系统有一些共同的特征，但它们的一个关键区别是它们的动态。深度网络通常在串行阶段中操作，其中每个层在后续层中开始处理之前完成其计算。相比之下，生物系统具有级联动力学：信息从所有层的神经元并行传播，但传输随着时间的推移逐渐发生，即使在前馈架构中也会导致速度-精度权衡。我们通过构建级联ResNet来探索生物启发并行硬件的后果，其中每个残差块都有传播延迟，但所有块都以有状态的方式并行更新。由于通过跳过连接传输的信息避免了延迟，因此架构的功能深度会随着时间的推移而增加，从而产生随内部处理时间而改进的随时预测。我们引入了一种时间差训练损失，它实现了严格优于标准损失的速度-精度曲线，并使级联架构能够超越最先进的任何时间预测方法。级联架构具有有趣的特性，包括：它比非典型实例更快地对典型实例进行分类;它比传统的ResNet对持久和瞬态噪声更鲁棒;其随时间变化的输出轨迹提供了一个可用于改善信息处理和推理的信号。
英文摘要：
Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce a temporal-difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
