This paper introduces and evaluates a novel training method for neural networks: Dual Variable Learning Rates (DVLR). Building on insights from behavioral psychology, the dual learning rates are used to emphasize correct and incorrect responses differently, thereby making the feedback to the network more specific. Further, the learning rates are varied as a function of the network's performance, thereby making it more efficient. DVLR was implemented on three types of networks: feedforward, convolutional, and residual, and two domains: MNIST and CIFAR-10. The results suggest a consistently improved accuracy, demonstrating that DVLR is a promising, psychologically motivated technique for training neural network models.
Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations
Cornell University(康奈尔大学);Cornell Tech(康奈尔科技学院)
我们提出了精确门控(PG),这是一种用于深度神经网络的端到端可训练动态双精度量化技术。PG以低精度计算大多数特征,仅以较高精度计算一小部分重要特征以保持准确性。所提出的方法适用于各种DNN架构,并显着降低了DNN执行的计算成本,几乎没有精度损失。我们的实验表明,PG在CNN上取得了优异的结果,包括静态压缩的移动友好网络,如ShuffleNet。与最先进的基于预测的量化方案相比,PG在ImageNet上实现了相同或更高的精度,但计算量减少了2.4 $\times$。PG也适用于RNN。与8位均匀量化相比,PG在Penn Tree Bank数据集上的LSTM上每字的困惑度提高了1.2%,计算成本降低了2.7 $\times$。代码可在:https://github.com/cornell-zhang/dnn-gating
Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers
Joint Graduate School of Mathematics for Innovation(联合创新数学研究生院);Kyushu University(九州大学);Faculty of Information Science and Electrical Engineering(信息科学与电气工程学院);Université Savoie Mont Blanc(萨瓦大学)
Improving Information Cascade Modeling by Social Topology and Dual Role User Dependency
Fudan University(复旦大学)
社交网络上的信息扩散(也称为信息级联)由于其在许多领域的应用价值,在过去的十年中得到了大量的研究。近年来,包括递归神经网络模型在内的序列模型被广泛应用于信息级联的预测。然而,顺序模型捕获的级联序列中的用户依赖性通常是单向的,并且与扩散树不一致。例如,后继者的真实触发可以是非直接前导者而不是序列中的直接前导者。为了更充分地捕捉用户依赖关系,实现精确的级联建模,提出了一种非顺序信息级联模型TAN-DRUD(Topology-aware Attention Networks with Dual Role User Dependency)。受经典通信理论的启发,TAN-DRUD通过捕捉信息发送者和接收者的双重角色用户依赖关系,在信息级联建模方面取得了令人满意的性能。此外,TANDRUD将社交拓扑纳入两级注意力网络中,以增强信息扩散预测。我们在三个级联数据集上的大量实验表明,我们的模型不仅优于最先进的级联模型,而且能够利用拓扑信息和推断扩散树。
Pareto-Improving Adversarial Attacks with Primal-Dual Regularization
National University of Defense Technology(国防科技大学);Academy of Military Science(军事科学院);Sun Yat-sen University(中山大学);Hefei Institute of Technology(合肥工业大学)
Dual-CBA: Improving Online Continual Learning via Dual Continual Bias Adaptors from a Bi-level Optimization Perspective
School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学院);City University of Hong Kong(香港城市大学);School of Mathematics and Statistics, Xidian University(西安电子科技大学数学与统计学院)
2DXformer: Dual Transformers for Wind Power Forecasting with Dual Exogenous Variables
School of Artificial Intelligence, Hebei University of Technology, Tianjin, China(人工智能学院,河北工业大学,天津,中国);State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin, China(电力设备可靠性与智能化国家重点实验室,河北工业大学,天津,中国);Hebei Province Key Laboratory of Big Data Calculation, Hebei University of Technology, Tianjin, China(河北省大数据计算重点实验室,河北工业大学,天津,中国)
Fast State-Augmented Learning for Wireless Resource Allocation with Dual Variable Regression
Department of Electrical and Systems Engineering, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系);Department of Biostatistics and Bioinformatics, Duke University(杜克大学生物统计与生物信息学系);Johns Hopkins Applied Research Laboratory (APL)(约翰霍普金斯应用研究实验室)
Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network
School of Information Science and Engineering, Lanzhou University(兰州大学信息科学与工程学院);College of Computer Science and Engineering, Shandong University of Science and Technology(山东科技大学计算机科学与工程学院)
AirCast: Improving Air Pollution Forecasting Through Multi-Variable Data Alignment
Wageningen University and Research(瓦赫宁根大学与研究中心);Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);Weizmann Institute of Science(魏茨曼科学研究所)
Efficiently improving key weather variables forecasting by performing the guided iterative prediction in latent space
Wuhan University(武汉大学);Hubei Key Laboratory of Quantitative Remote Sensing of Land and Atmosphere(湖北省陆气定量遥感重点实验室);School of Remote Sensing and Information Engineering(遥感信息工程学院);Hubei Luojia Laboratory(湖北珞珈实验室)