Quantized deep neural networks (QDNNs) are necessary for low-power, high throughput, and embedded applications. Previous studies mostly focused on developing optimization methods for the quantization of given models. However, quantization sensitivity depends on the model architecture. Therefore, the model selection needs to be a part of the QDNN design process. Also, the characteristics of weight and activation quantization are quite different. This study proposes a holistic approach for the optimization of QDNNs, which contains QDNN training methods as well as quantization-friendly architecture design. Synthesized data is used to visualize the effects of weight and activation quantization. The results indicate that deeper models are more prone to activation quantization, while wider models improve the resiliency to both weight and activation quantization. This study can provide insight into better optimization of QDNNs.
Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coefficient, and the size of teacher network for QDNN training. We analyze the effect of these hyper-parameters for QDNN optimization with KD. We find that these hyper-parameters are inter-related, and also introduce a simple and effective technique that reduces \textit{coefficient} duri…
In this paper, we develop a new algorithm, Annealed Skewed SGD - AskewSGD - for training deep neural networks (DNNs) with quantized weights. First, we formulate the training of quantized neural networks (QNNs) as a smoothed sequence of interval-constrained optimization problems. Then, we propose a new first-order stochastic method, AskewSGD, to solve each constrained optimization subproblem. Unlike algorithms with active sets and feasible directions, AskewSGD avoids projections or optimization under the entire feasible set and allows iterates that are infeasible. The numerical complexity of As…
Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in two-layer neural networks from the recent lit…
OMAC: A Holistic Optimization Framework for LLM-Based Multi-Agent Collaboration
Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, United States（得克萨斯大学奥斯汀分校电子与计算机工程系）；Intuit AI Research, Mountain View, United States（Intuit AI研究）
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Hi Lab, Xiaohongshu Inc.（小红书公司Hi实验室）；Project Numina（项目Numina）；School of Data Science, Fudan University（复旦大学数据科学学院）
Amazon Web Services, Inc（亚马逊云科技）；Scroll Foundation（Scroll基金会）；OctoML, Inc（OctoML公司）
As deep learning is pervasive in modern applications, many deep learning frameworks are presented for deep learning practitioners to develop and train DNN models rapidly. Meanwhile, as training large deep learning models becomes a trend in recent years, the training throughput and memory footprint are getting crucial. Accordingly, optimizing training workloads with compiler optimizations is inevitable and getting more and more attentions. However, existing deep learning compilers (DLCs) mainly target inference and do not incorporate holistic optimizations, such as automatic differentiation and…
Yale University（耶鲁大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Pennsylvania（宾夕法尼亚大学）
How can we efficiently mitigate the overhead of gradient communications in distributed optimization? This problem is at the heart of training scalable machine learning models and has been mainly studied in the unconstrained setting. In this paper, we propose Quantized-Frank-Wolfe (QFW), the first projection-free and communication-efficient algorithm for solving constrained optimization problems at scale. We consider both convex and non-convex objective functions, expressed as a finite-sum or more generally a stochastic optimization problem, and provide strong theoretical guarantees on the conv…
Machine learning workflow development is a process of trial-and-error: developers iterate on workflows by testing out small modifications until the desired accuracy is achieved. Unfortunately, existing machine learning systems focus narrowly on model training---a small fraction of the overall development time---and neglect to address iterative development. We propose Helix, a machine learning system that optimizes the execution across iterations---intelligently caching and reusing, or recomputing intermediates as appropriate. Helix captures a wide variety of application needs within its Scala …
RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Systems Research Institute, Polish Academy of Sciences（波兰科学院系统研究所）；Artificial Intelligence Research Institute (AIRI)（人工智能研究院）
We present a novel procedure for optimization based on the combination of efficient quantized tensor train representation and a generalized maximum matrix volume principle. We demonstrate the applicability of the new Tensor Train Optimizer (TTOpt) method for various tasks, ranging from minimization of multidimensional functions to reinforcement learning. Our algorithm compares favorably to popular evolutionary-based methods and outperforms them by the number of function evaluations or execution time, often by a significant margin.
Stanford University（斯坦福大学）；Sharif University of Technology（谢里夫理工大学）；Advanced Communications Research Institute (ACRI)（高级通信研究所）
In this paper, we introduce a novel and robust approach to Quantized Matrix Completion (QMC). First, we propose a rank minimization problem with constraints induced by quantization bounds. Next, we form an unconstrained optimization problem by regularizing the rank function with Huber loss. Huber loss is leveraged to control the violation from quantization bounds due to two properties: 1- It is differentiable, 2- It is less sensitive to outliers than the quadratic loss. A Smooth Rank Approximation is utilized to endorse lower rank on the genuine data matrix. Thus, an unconstrained optimization…
University of Oklahoma（俄克拉荷马大学）；Qatar University（卡塔尔大学）
In the wake of network densification and multi-band operation in emerging cellular networks, mobility and handover management is becoming a major bottleneck. The problem is further aggravated by the fact that holistic mobility management solutions for different types of handovers, namely inter-frequency and intra-frequency handovers, remain scarce. This paper presents a first mobility management solution that concurrently optimizes inter-frequency related A5 parameters and intra-frequency related A3 parameters. We analyze and optimize five parameters namely A5-time to trigger (TTT), A5-thresho…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；University of Pennsylvania（宾夕法尼亚大学）
We consider the problem of decentralized consensus optimization, where the sum of $n$ smooth and strongly convex functions are minimized over $n$ distributed agents that form a connected network. In particular, we consider the case that the communicated local decision variables among nodes are quantized in order to alleviate the communication bottleneck in distributed optimization. We propose the Quantized Decentralized Gradient Descent (QDGD) algorithm, in which nodes update their local decision variables by combining the quantized information received from their neighbors with their local in…
University of Kaiserslautern（凯泽斯劳滕大学）；Fraunhofer ITWM（弗劳恩霍夫工业数学研究所）
Deep Neural Networks (DNNs) are capable of solving complex problems in domains related to embedded systems, such as image and natural language processing. To efficiently implement DNNs on a specific FPGA platform for a given cost criterion, e.g. energy efficiency, an enormous amount of design parameters has to be considered from the topology down to the final hardware implementation. Interdependencies between the different design layers have to be taken into account and explored efficiently, making it hardly possible to find optimized solutions manually. An automatic, holistic design approach …
Designing a deep neural network (DNN) with good generalization capability is a complex process especially when the weights are severely quantized. Model averaging is a promising approach for achieving the good generalization capability of DNNs, especially when the loss surface for training contains many sharp minima. We present a new quantized neural network optimization approach, stochastic quantized weight averaging (SQWA), to design low-precision DNNs with good generalization capability using model averaging. The proposed approach includes (1) floating-point model training, (2) direct quant…
IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Department of Systems Engineering；Engineering Management, The Chinese University of Hong Kong, Hong Kong, China；DAMO Academy, Alibaba Group, Hangzhou, China；Hong Kong University of Science；Hupan Lab, Hangzhou, China；Department of Computer Science, Hong Kong Baptist University, Hong Kong, China；School of Artificial Intelligence, Shenzhen University, Shenzhen, China；Tencent, Shenzhen, China
GQFedWAvg: Optimization-Based Quantized Federated Learning in General Edge Computing Systems
School of EIEE, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；IoT Thrust, the HKUST (GZ)（香港科技大学（广州）物联网枢纽）；Department of ECE, the HKUST（香港科技大学电子与计算机工程学系）
We consider the problem of deep neural net compression by quantization: given a large, reference net, we want to quantize its real-valued weights using a codebook with $K$ entries so that the training loss of the quantized net is minimal. The codebook can be optimally learned jointly with the net, or fixed, as for binarization or ternarization approaches. Previous work has quantized the weights of the reference net, or incorporated rounding operations in the backpropagation algorithm, but this has no guarantee of converging to a loss-optimal, quantized net. We describe a new approach based on …
Decentralized Optimization on Compact Submanifolds by Quantized Riemannian Gradient Tracking
National Special Education Resource Center for Children with Autism, Zhejiang Normal University（浙江师范大学生自闭症教育资源中心）；School of Computer Science and Technology, Zhejiang Normal University（浙江师范大学计算机科学与技术学院）；China Mobile Research Institute（中国移动研究院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制系统研究所）；SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University（浙江智能教育技术与应用重点实验室）；School of Information Engineering, Huzhou University（湖州大学信息工程学院）；Centre for Frontier Artificial Intelligence Research, A*STAR（A*STAR前沿人工智能研究中心）
Institute of Science and Technology Austria (ISTA)（奥地利科学技术研究所（ISTA））；University of Surrey（萨里大学）；ETH Zurich（苏黎世联邦理工学院）
Decentralized optimization is emerging as a viable alternative for scalable distributed machine learning, but also introduces new challenges in terms of synchronization costs. To this end, several communication-reduction techniques, such as non-blocking communication, quantization, and local steps, have been explored in the decentralized setting. Due to the complexity of analyzing optimization in such a relaxed setting, this line of work often assumes \emph{global} communication rounds, which require additional synchronization. In this paper, we consider decentralized optimization in the simpl…
Due to its efficiency and ease to implement, stochastic gradient descent (SGD) has been widely used in machine learning. In particular, SGD is one of the most popular optimization methods for distributed learning. Recently, quantized SGD (QSGD), which adopts quantization to reduce the communication cost in SGD-based distributed learning, has attracted much attention. Although several QSGD methods have been proposed, some of them are heuristic without theoretical guarantee, and others have high quantization variance which makes the convergence become slow. In this paper, we propose a new method…
Vector Quantized-Elites: Unsupervised and Problem-Agnostic Quality-Diversity Optimization
Computational Intelligence Laboratory (CILab), Department of Mathematics, University of Patras（数学系计算智能实验室（CILab），帕特拉大学）；Laboratory of Automation and Robotics (LAR) in the Department of Electrical & Computer Engineering, University of Patras（电气与计算机工程系自动化与机器人实验室（LAR），帕特拉大学）
University of California, Irvine（加州大学欧文分校）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
Deep neural networks (DNNs) are quantized for efficient inference on resource-constrained platforms. However, training deep learning models with low-precision weights and activations involves a demanding optimization task, which calls for minimizing a stage-wise loss function subject to a discrete set-constraint. While numerous training methods have been proposed, existing studies for full quantization of DNNs are mostly empirical. From a theoretical point of view, we study practical techniques for overcoming the combinatorial nature of network quantization. Specifically, we investigate a simp…
Low-bit width neural networks have been extensively explored for deployment on edge devices to reduce computational resources. Existing approaches have focused on gradient-based optimization in a two-stage train-and-compress setting or as a combined optimization where gradients are quantized during training. Such schemes require high-performance hardware during the training phase and usually store an equivalent number of full-precision weights apart from the quantized weights. In this paper, we explore methods of direct combinatorial optimization in the problem of risk minimization with binary…
Duke University（杜克大学）；Monta Vista High School（蒙塔维斯塔高中）
Neural networks are getting better accuracy with higher energy and computational cost. After quantization, the cost can be greatly saved, and the quantized models are more hardware friendly with acceptable accuracy loss. On the other hand, recent research has found that neural networks are vulnerable to adversarial attacks, and the robustness of a neural network model can only be improved with defense methods, such as adversarial training. In this work, we find that adversarially-trained neural networks are more vulnerable to quantization loss than plain models. To minimize both the adversaria…
Optimal algorithm design for federated learning (FL) remains an open problem. This paper explores the full potential of FL in practical edge computing systems where workers may have different computation and communication capabilities, and quantized intermediate model updates are sent between the server and workers. First, we present a general quantized parallel mini-batch stochastic gradient descent (SGD) algorithm for FL, namely GenQSGD, which is parameterized by the number of global iterations, the numbers of local iterations at all workers, and the mini-batch size. We also analyze its conv…
Exploration of Activation Fault Reliability in Quantized Systolic Array-Based DNN Accelerators
Tallinn University of Technology（塔林理工大学）；University of Alberta（阿尔伯塔大学）；Shahid Bahonar University of Kerman（克尔曼沙希德·巴霍纳尔大学）；York University（约克大学）；Mälardalen University（梅拉达伦大学）
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with {\em diverse resources}. In this work, we introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeL that facilitates collective training with heterogeneous clients while respecting resource diversity. For personalization, we allow clients to learn \textit{compressed personalized models} with different quantization parameters…
With the proliferation of deep convolutional neural network (CNN) algorithms for mobile processing, limited precision quantization has become an essential tool for CNN efficiency. Consequently, various works have sought to design fixed precision quantization algorithms and quantization-focused optimization techniques that minimize quantization induced performance degradation. However, there is little concrete understanding of how various CNN design decisions/best practices affect quantized inference behaviour. Weight initialization strategies are often associated with solving issues such as va…
