In this work, we investigate the fundamental trade-off regarding accuracy and parameter efficiency in the parameterization of neural network weights using predictor networks. We present a surprising finding that, when recovering the original model accuracy is the sole objective, it can be achieved effectively through the weight reconstruction objective alone. Additionally, we explore the underlying factors for improving weight reconstruction under parameter-efficiency constraints, and propose a novel training scheme that decouples the reconstruction objective from auxiliary objectives such as knowledge distillation that leads to significant improvements compared to state-of-the-art approaches. Finally, these results pave way for more practical scenarios, where one needs to achieve improvements on both model accuracy and predictor network parameter-efficiency simultaneously.
Layer-Guided UAV Tracking: Enhancing Efficiency and Occlusion Robustness
Department of Control Science and Engineering, University of Shanghai for Science and Technology(上海理工大学控制科学与工程学院);Busness School, University of Shanghai for Science and Technology(上海理工大学商学院)
Enhancing the Parameterization of Reservoir Properties for Data Assimilation Using Deep VAE-GAN
Departamento de Engenharia de Minas e de Petróleo, Escola Politécnica, Universidade de São Paulo(圣保罗大学采矿与石油工程系,理工学院);Department of Earth Science and Engineering, Imperial College London(伦敦帝国理工学院地球科学与工程系)
Mitigating severe over-parameterization in deep convolutional neural networks through forced feature abstraction and compression with an entropy-based heuristic
Auckland University of Technology(奥克兰理工大学)
卷积神经网络(CNN),如ResNet-50,DenseNet-40和ResNeXt-56是严重过度参数化的,因此需要增加模型训练所需的计算资源,模型训练以指数方式增加模型深度。在本文中,我们提出了一个基于熵的卷积层估计(EBCLE)启发式算法,这是强大的,简单的,但有效地解决了过度参数化的问题,关于CNN模型的网络深度。EBCLE启发式算法采用输入数据集的熵数据分布的先验知识来确定卷积网络深度的上限,超过该上限,恒等变换是普遍的,为增强模型性能提供了微不足道的贡献。通过强制特征压缩和抽象来限制深度冗余,限制了过度参数化,同时减少了24.99% - 78.59%的训练时间,而不会降低模型性能。我们提出了经验证据来强调使用EBCLE启发式训练的较宽但较浅的模型的相对有效性,该模型保持或优于较窄但较深的模型的基线分类准确性。EBCLE启发式算法在架构上是不可知的,并且基于EBCLE的CNN模型限制深度冗余,从而增强了对可用计算资源的利用。提出的EBCLE启发式是一种令人信服的技术,研究人员可以分析证明他们对CNN的HyperParameter(HP)选择。EBCLE启发式训练CNN模型的经验验证是在五个基准数据集(ImageNet 32,CIFAR-10/100,STL-10,MNIST)和四个网络架构(DenseNet,ResNet,ResNeXt和EfficientNet B 0-…
Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST));Pohang University of Science and Technology (POSTECH)(浦项科技大学(POSTECH));OMELET
Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
University of California, Berkeley(加州大学伯克利分校);Technical University of Munich(慕尼黑工业大学);California Institute of Technology(加州理工学院);Virginia Tech(弗吉尼亚理工大学)
Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
Georgia Institute of Technology(佐治亚理工学院);National Taiwan University(国立台湾大学);University of Palermo(帕尔米奥大学);NVIDIA Research(NVIDIA研究);Hon-Hai Quantum Computing Research Center(鸿海量子计算研究中心)
Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
Amazon/A9(亚马逊/A9)
梯度消失和梯度爆炸是训练深度神经网络的两个主要障碍,特别是在捕获递归神经网络~(RNN)中的长程依赖关系方面。在本文中,我们提出了RNN的转移矩阵的有效参数化,使我们能够稳定其训练中出现的梯度。具体来说,我们通过奇异值分解(SVD)来参数化转移矩阵,这允许我们显式地跟踪和控制其奇异值。我们通过使用数值线性代数中常见的工具来实现效率,即Householder反射器用于表示SVD中出现的正交矩阵。通过显式控制奇异值,我们提出的Spectral-RNN方法使我们能够轻松解决梯度爆炸问题,并且我们观察到它在很大程度上凭经验解决了梯度消失问题。我们注意到SVD参数化可以用于任何矩形权重矩阵,因此它可以很容易地扩展到任何深度神经网络,例如多层感知器。从理论上讲,我们证明了我们的参数化不会失去任何表达能力,并展示了它如何控制RNN的分类任务的泛化。%,并展示它如何潜在地使优化过程更容易。我们广泛的实验结果也表明,所提出的框架收敛速度更快,具有良好的泛化能力,特别是在捕捉长范围的依赖关系,如合成添加和复制任务,以及MNIST和Penn Tree Bank数据集。
TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency
School of Computing, Southern Illinois University(南伊利诺伊大学计算机学院);Security, Privacy and Intelligence for Edge Devices Laboratory (SPEED Lab)(边缘设备安全、隐私与智能实验室(SPEED Lab))