首页/论文/1802.03488

arXiv 1802.034882018-02-10stat.MLcs.LG

Generalization of an Upper Bound on the Number of Nodes Needed to Achieve Linear Separability

Radboud University（拉德堡德大学）；Donders Institute for Brain, Cognition and Behaviour（东德斯脑、认知与行为研究所）

Marjolein Troost, Katja Seeliger, Marcel van Gerven

备注：: Presented at the 29th Benelux Conference on Artificial Intelligence (BNAIC 2017)

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

An important issue in neural network research is how to choose the number of nodes and layers such as to solve a classification problem. We provide new intuitions based on earlier results by An et al. (2015) by deriving an upper bound on the number of nodes in networks with two hidden layers such that linear separability can be achieved. Concretely, we show that if the data can be described in terms of N finite sets and the used activation function f is non-constant, increasing and has a left asymptote, we can derive how many nodes are needed to linearly separate these sets. This will be an upper bound that depends on the structure of the data. This structure can be analyzed using an algorithm. For the leaky rectified linear activation function, we prove separately that under some conditions on the slope, the same number of layers and nodes as for the aforementioned activation functions is sufficient. We empirically validate our claims.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization

University of Waterloo（滑铁卢大学）

Recently there has been increased interest in semi-supervised classification in the presence of graphical information. A new class of learning models has emerged that relies, at its most basic level, on classifying the data after first applying a graph convolution. To understand the merits of this approach, we study the classification of a mixture of Gaussians, where the data corresponds to the node attributes of a stochastic block model. We show that graph convolution extends the regime in which the data is linearly separable by a factor of roughly $1/\sqrt{D}$, where $D$ is the expected degr…

2022-02-04cs.LGstat.ML

SepNE: Bringing Separability to Network Embedding

Yuanpei College, Peking University（北京大学元培学院）；Key Laboratory of Machine Perception, Ministry of Education, Peking University（北京大学机器感知与智能教育部重点实验室）

Many successful methods have been proposed for learning low dimensional representations on large-scale networks, while almost all existing methods are designed in inseparable processes, learning embeddings for entire networks even when only a small proportion of nodes are of interest. This leads to great inconvenience, especially on super-large or dynamic networks, where these methods become almost impossible to implement. In this paper, we formalize the problem of separated matrix factorization, based on which we elaborate a novel objective function that preserves both local and global inform…

2019-02-26cs.LGcs.SIstat.ML

Neural Decision Trees

Rice University（莱斯大学）

In this paper we propose a synergistic melting of neural networks and decision trees (DT) we call neural decision trees (NDT). NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more than one layer is used. This way, each MLP can be seen as a node of the tree. We then show that with the weight sharing asumption among those units, we end up with a Hashing Neural Network (HNN) which is a multilayer perceptron with sigmoid activation function for the last layer…

2017-03-06cs.LGstat.ML

理解图神经网络的异质性

Understanding Heterophily for Graph Neural Networks

Beihang University（北京航空航天大学）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Shen Yuan Honors College, Beihang University（北京航空航天大学沈元荣誉学院）；School of Artificial Intelligence, Hebei University of Technology（河北工业大学人工智能学院）

本文提出异质性随机块模型（HSBM），从理论上分析了不同异质性模式对图神经网络的影响，证明可分离性增益由邻域分布距离与平均节点度共同决定，拓扑噪声会降低有效节点度，且多层图卷积在广泛条件下仍能保持节点可分离性。

2024-06-04cs.LGstat.ML

鲁棒对比图聚类与自适应局部-全局整合

Robust Contrastive Graph Clustering with Adaptive Local-Global Integration

School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；College of Management and Economics, Tianjin University（天津大学管理学院）

提出一种对比图聚类框架，通过注意力机制自适应融合多尺度局部结构和全局语义原型，以解决复杂图中高阶局部结构捕获不足和全局语义忽略问题，提升聚类性能。

2026-05-30cs.LG

Coloring graph neural networks for node disambiguation

École Polytechnique（巴黎综合理工学院）；Noah’s Ark Lab, Huawei（华为诺亚方舟实验室）

In this paper, we show that a simple coloring scheme can improve, both theoretically and empirically, the expressive power of Message Passing Neural Networks(MPNNs). More specifically, we introduce a graph neural network called Colored Local Iterative Procedure (CLIP) that uses colors to disambiguate identical node attributes, and show that this representation is a universal approximator of continuous functions on graphs with node attributes. Our method relies on separability , a key topological characteristic that allows to extend well-chosen neural networks into universal representations. Fi…

2019-12-12cs.LGstat.ML

更多相关论文收起更多论文

使用小型储层重构混沌吸引子——拓扑结构的影响

Chaotic attractor reconstruction using small reservoirs -- the influence of topology

Technische Universität Ilmenau（伊尔梅瑙工业大学）

本研究通过小型非耦合节点储层改进混沌吸引子重构，发现其优于复杂拓扑，且与更小谱半径相关，有利于硬件实现。

2024-02-23cs.ETcs.LGmath-ph

Angular separability of data clusters or network communities in geometrical space and its relevance to hyperbolic embedding

Technische Universität Dresden（德累斯顿工业大学）；Tsinghua University（清华大学）

Analysis of 'big data' characterized by high-dimensionality such as word vectors and complex networks requires often their representation in a geometrical space by embedding. Recent developments in machine learning and network geometry have pointed out the hyperbolic space as a useful framework for the representation of this data derived by real complex physical systems. In the hyperbolic space, the radial coordinate of the nodes characterizes their hierarchy, whereas the angular distance between them represents their similarity. Several studies have highlighted the relationship between the an…

2019-06-28cs.LGcs.SIphysics.soc-ph

Mapping Energy Landscapes of Non-Convex Learning Problems

University of California, Los Angeles（加州大学洛杉矶分校）；ShanghaiTech University（上海科技大学）

In many statistical learning problems, the target functions to be optimized are highly non-convex in various model spaces and thus are difficult to analyze. In this paper, we compute \emph{Energy Landscape Maps} (ELMs) which characterize and visualize an energy function with a tree structure, in which each leaf node represents a local minimum and each non-leaf node represents the barrier between adjacent energy basins. The ELM also associates each node with the estimated probability mass and volume for the corresponding energy basin. We construct ELMs by adopting the generalized Wang-Landau al…

2014-10-02cs.LGstat.ML

Graph-Bert: Only Attention is Needed for Learning Graph Representations

IFM Lab（IFM实验室）；Florida State University（佛罗里达州立大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Beijing University of Posts and Telecommunications（北京邮电大学）

The dominant graph neural networks (GNNs) over-rely on the graph links, several serious performance problems with which have been witnessed already, e.g., suspended animation problem and over-smoothing problem. What's more, the inherently inter-connected nature precludes parallelization within the graph, which becomes critical for large-sized graph, as memory constraints limit batching across the nodes. In this paper, we will introduce a new graph neural network, namely GRAPH-BERT (Graph based BERT), solely based on the attention mechanism without any graph convolution or aggregation operators…

2020-01-22cs.LGcs.NEstat.ML

Parallel Coordinate Descent Methods for Big Data Optimization

University of Edinburgh（爱丁堡大学）

In this work we show that randomized (block) coordinate descent methods can be accelerated by parallelization when applied to the problem of minimizing the sum of a partially separable smooth convex function and a simple separable convex function. The theoretical speedup, as compared to the serial method, and referring to the number of iterations needed to approximately solve the problem with high probability, is a simple expression depending on the number of parallel processors and a natural and easily computable measure of separability of the smooth component of the objective function. In th…

2013-11-25cs.AImath.OCstat.ML

A case where a spindly two-layer linear network whips any neural network with a fully connected input layer

Google Research（谷歌研究院）；Poznan University of Technology（波兹南理工大学）

It was conjectured that any neural network of any structure and arbitrary differentiable transfer functions at the nodes cannot learn the following problem sample efficiently when trained with gradient descent: The instances are the rows of a $d$-dimensional Hadamard matrix and the target is one of the features, i.e. very sparse. We essentially prove this conjecture: We show that after receiving a random training set of size $k < d$, the expected square loss is still $1-\frac{k}{(d-1)}$. The only requirement needed is that the input layer is fully connected and the initial weight vectors of th…

2020-10-16cs.LGstat.ML

Understanding and Resolving Performance Degradation in Graph Convolutional Networks

National University of Singapore（新加坡国立大学）；PayPal Innovation Lab（PayPal创新实验室）；Alibaba Group（阿里巴巴集团）

A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data. Though powerful, GCNs tend to suffer performance drop when the model gets deep. Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated. In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works. We find that TRANs contribute significantly, or even more than PROPs, t…

2021-09-13cs.LGstat.ML

Estimating Node Importance in Knowledge Graphs Using Graph Neural Networks

Carnegie Mellon University（卡内基梅隆大学）；Amazon（亚马逊）

How can we estimate the importance of nodes in a knowledge graph (KG)? A KG is a multi-relational graph that has proven valuable for many tasks including question answering and semantic search. In this paper, we present GENI, a method for tackling the problem of estimating node importance in KGs, which enables several downstream applications such as item recommendation and resource allocation. While a number of approaches have been developed to address this problem for general graphs, they do not fully utilize information available in KGs, or lack flexibility needed to model complex relationsh…

2019-06-16cs.IRcs.LGstat.ML

稀疏分布式学习 via 异质扩散自适应网络

Sparse Distributed Learning via Heterogeneous Diffusion Adaptive Networks

本文提出通过异质扩散自适应网络实现稀疏参数向量的分布式估计，通过选择性应用凸正则化方法减少计算开销，同时保持最优性能。

2014-10-26cs.DCcs.LGcs.SY