Graph neural networks (GNNs) have been widely used to predict properties and heuristics of mixed-integer linear programs (MILPs) and hence accelerate MILP solvers. This paper investigates the capacity of GNNs to represent strong branching (SB), the most effective yet computationally expensive heuristic employed in the branch-and-bound algorithm. In the literature, message-passing GNN (MP-GNN), as the simplest GNN structure, is frequently used as a fast approximation of SB and we find that not all MILPs's SB can be represented with MP-GNN. We precisely define a class of "MP-tractable" MILPs for which MP-GNNs can accurately approximate SB scores. Particularly, we establish a universal approximation theorem: for any data distribution over the MP-tractable class, there always exists an MP-GNN that can approximate the SB score with arbitrarily high accuracy and arbitrarily high probability, which lays a theoretical foundation of the existing works on imitating SB with MP-GNN. For MILPs without the MP-tractability, unfortunately, a similar result is impossible, which can be illustrated by two MILP instances with different SB scores that cannot be distinguished by any MP-GNN, regardless of the number of parameters. Recognizing this, we explore another GNN structure called the second-order folklore GNN (2-FGNN) that overcomes this limitation, and the aforementioned universal approximation theorem can be extended to the entire MILP space using 2-FGNN, regardless of the MP-tractability. A small-scale numerical experiment is conducted to directly validate our theoretical findings.
The graph Laplacian regularization term is usually used in semi-supervised representation learning to provide graph structure information for a model $f(X)$. However, with the recent popularity of graph neural networks (GNNs), directly encoding graph structure $A$ into a model, i.e., $f(A, X)$, has become the more common approach. While we show that graph Laplacian regularization brings little-to-no benefit to existing GNNs, and propose a simple but non-trivial variant of graph Laplacian regularization, called Propagation-regularization (P-reg), to boost the performance of existing GNN models.…
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；University of Pittsburgh（匹兹堡大学）；University of Maryland, College Park（马里兰大学学院公园分校）；University of Connecticut（康涅狄格大学）
Rethinking Neural Combinatorial Optimization for Vehicle Routing Problems with Different Constraint Tightness Degrees
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China（自动化与智能制造学院，南方科技大学，深圳，中国）；Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, Southern University of Science and Technology, Shenzhen, China（广东省全驱动系统控制理论与技术重点实验室，南方科技大学，深圳，中国）；Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands（工业工程与创新科学系，埃因霍温理工大学，埃因霍温，荷兰）；School of Computing, National University of Singapore, Singapore（计算学院，新加坡国立大学）
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
College of Computer Science, Sichuan University, Chengdu, China（四川大学计算机学院，成都，中国）；Institute of High Performance Computing, A STAR, Singapore（高性能计算研究院，新加坡）
Rethinking Neural Width for Alternating Current Optimal Power Flow Proxies
Department of Electrical Engineering, National Institute of Technology Kurukshetra, India（印度克什米尔国立理工学院电气工程系）；Indraprastha Institute of Information Technology Delhi, India（印度德里印度理工信息学院）；Department of Electrical Engineering, Indian Institute of Technology Roorkee, India（印度罗尔基印度理工学院电气工程系）
Calculation Consulting（计算咨询公司）；ICSI（国际计算机科学研究所）；University of California at Berkeley（加州大学伯克利分校）
We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks. The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks. Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input),…
One of the most critical problems in weight-sharing neural architecture search is the evaluation of candidate models within a predefined search space. In practice, a one-shot supernet is trained to serve as an evaluator. A faithful ranking certainly leads to more accurate searching results. However, current methods are prone to making misjudgments. In this paper, we prove that their biased evaluation is due to inherent unfairness in the supernet training. In view of this, we propose two levels of constraints: expectation fairness and strict fairness. Particularly, strict fairness ensures equal…
Despite multiple efforts made towards adopting complex-valued deep neural networks (DNNs), it remains an open question whether complex-valued DNNs are generally more effective than real-valued DNNs for monaural speech enhancement. This work is devoted to presenting a critical assessment by systematically examining complex-valued DNNs against their real-valued counterparts. Specifically, we investigate complex-valued DNN atomic units, including linear layers, convolutional layers, long short-term memory (LSTM), and gated linear units. By comparing complex- and real-valued versions of fundamenta…
Video action recognition has been partially addressed by the CNNs stacking of fixed-size 3D kernels. However, these methods may under-perform for only capturing rigid spatial-temporal patterns in single-scale spaces, while neglecting the scale variances across different action primitives. To overcome this limitation, we propose to learn the optimal-scale kernels from the data. More specifically, an \textit{action perceptron synthesizer} is proposed to generate the kernels from a bag of fixed-size kernels that are interacted by dense routing paths. To guarantee the interaction richness and the …
Rethinking Convolutional Networks for Attribute-Aware Sequential Recommendation
Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany（希尔德斯海姆大学信息系统与机器学习实验室）；VWFS Data Analytics Research Center (VWFS DARC), University of Hildesheim, Germany（希尔德斯海姆大学VWFS数据分析研究中心）；Volkswagen Financial Services AG, Germany（德国大众金融服务有限公司）
We utilize a connection between compositional kernels and branching processes via Mehler's formula to study deep neural networks. This new probabilistic insight provides us a novel perspective on the mathematical role of activation functions in compositional neural networks. We study the unscaled and rescaled limits of the compositional kernels and explore the different phases of the limiting behavior, as the compositional depth increases. We investigate the memorization capacity of the compositional kernels and neural networks by characterizing the interplay among compositional depth, sample …
Rethinking Recurrent Neural Networks for Time Series Forecasting: A Reinforced Recurrent Encoder with Prediction-Oriented Proximal Policy Optimization
School of Management, Huazhong University of Science and Technology（华中科技大学管理学院）；Center for Applied Mathematics, École des Ponts ParisTech（巴黎高等理工学院应用数学中心）；School of Mechanical Engineering, Dalian Jiaotong University（大连交通大学机械工程学院）；School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University（虚拟现实技术与系统国家重点实验室，计算机科学与工程学院，北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Center for AI Business Innovation, Department of Management Science and Systems, School of Management, University at Buffalo（人工智能商业创新中心，管理科学与系统系，布法罗大学）
While much of the work in the design of convolutional networks over the last five years has revolved around the empirical investigation of the importance of depth, filter sizes, and number of feature channels, recent studies have shown that branching, i.e., splitting the computation along parallel but distinct threads and then aggregating their outputs, represents a new promising dimension for significant improvements in performance. To combat the complexity of design choices in multi-branch architectures, prior work has adopted simple strategies, such as a fixed branching factor, the same inp…
