Graph Transformers (GTs) have significantly advanced the field of graph representation learning by overcoming the limitations of message-passing graph neural networks (GNNs) and demonstrating promising performance and expressive power. However, the quadratic complexity of self-attention mechanism in GTs has limited their scalability, and previous approaches to address this issue often suffer from expressiveness degradation or lack of versatility. To address this issue, we propose AnchorGT, a novel attention architecture for GTs with global receptive field and almost linear complexity, which serves as a flexible building block to improve the scalability of a wide range of GT models. Inspired by anchor-based GNNs, we employ structurally important $k$-dominating node set as anchors and design an attention mechanism that focuses on the relationship between individual nodes and anchors, while retaining the global receptive field for all nodes. With its intuitive design, AnchorGT can easily replace the attention module in various GT models with different network architectures and structural encodings, resulting in reduced computational overhead without sacrificing performance. In addition, we theoretically prove that AnchorGT attention can be strictly more expressive than Weisfeiler-Lehman test, showing its superiority in representing graph structures. Our experiments on three state-of-the-art GT models demonstrate that their AnchorGT variants can achieve better results while being faster and significantly more memory efficient.
In recent studies, neural message passing has proved to be an effective way to design graph neural networks (GNNs), which have achieved state-of-the-art performance in many graph-based tasks. However, current neural-message passing architectures typically need to perform an expensive recursive neighborhood expansion in multiple rounds and consequently suffer from a scalability issue. Moreover, most existing neural-message passing schemes are inflexible since they are restricted to fixed-hop neighborhoods and insensitive to the actual demands of different nodes. We circumvent these limitations …
FISHDBC is a flexible, incremental, scalable, and hierarchical density-based clustering algorithm. It is flexible because it empowers users to work on arbitrary data, skipping the feature extraction step that usually transforms raw data in numeric arrays letting users define an arbitrary distance function instead. It is incremental and scalable: it avoids the $\mathcal O(n^2)$ performance of other approaches in non-metric spaces and requires only lightweight computation to update the clustering when few items are added. It is hierarchical: it produces a "flat" clustering which can be expanded …
This paper addresses the single-image compressive sensing (CS) and reconstruction problem. We propose a scalable Laplacian pyramid reconstructive adversarial network (LAPRAN) that enables high-fidelity, flexible and fast CS images reconstruction. LAPRAN progressively reconstructs an image following the concept of Laplacian pyramid through multiple stages of reconstructive adversarial networks (RANs). At each pyramid level, CS measurements are fused with a contextual latent vector to generate a high-frequency image residual. Consequently, LAPRAN can produce hierarchies of reconstructed images a…
Google AI（谷歌人工智能）；Courant Institute of Mathematical Sciences（库朗数学科学研究所）；Duke University（杜克大学）
AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention. Our framework is inspired by the AdaNet algorithm (Cortes et al., 2017) which learns the structure of a neural network as an ensemble of subnetworks. We designed it to: (1) integrate with the existing TensorFlow ecosystem, (2) offer sensible default search spaces to perform well on novel datasets, (3) present a flexible API to utilize expert information when available, and (4) efficiently accelerate training with distributed CPU, GPU, and T…
Massachusetts Institute of Technology（麻省理工学院）；University of Michigan（密歇根大学）
We present a novel intelligent tutoring system which builds upon well-established hypotheses in educational psychology and incorporates them inside of a scalable software architecture. Specifically, we build upon the known benefits of knowledge vocalization, parallel learning, and immediate feedback in the context of student learning. We show that open-source data combined with state-of-the-art techniques in deep learning and natural language processing can apply the benefits of these three factors at scale, while still operating at the granularity of individual student needs and recommendatio…
Azure Machine Learning（Azure 机器学习）；Microsoft New England Research and Development（微软新英格兰研发中心）
In this work we detail a novel open source library, called MMLSpark, that combines the flexible deep learning library Cognitive Toolkit, with the distributed computing framework Apache Spark. To achieve this, we have contributed Java Language bindings to the Cognitive Toolkit, and added several new components to the Spark ecosystem. In addition, we also integrate the popular image processing library OpenCV with Spark, and present a tool for the automated generation of PySpark wrappers from any SparkML estimator and use this tool to expose all work to the PySpark ecosystem. Finally, we provide …
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）
FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model
Shanghai AI Laboratory（上海人工智能实验室）；Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；University of Sydney（悉尼大学）；Tsinghua University（清华大学）
This paper presents Systolic-CNN, an OpenCL-defined scalable, run-time-flexible FPGA accelerator architecture, optimized for accelerating the inference of various convolutional neural networks (CNNs) in multi-tenancy cloud/edge computing. The existing OpenCL-defined FPGA accelerators for CNN inference are insufficient due to limited flexibility for supporting multiple CNN models at run time and poor scalability resulting in underutilized FPGA resources and limited computational parallelism. Systolic-CNN adopts a highly pipelined and paralleled 1-D systolic array architecture, which efficiently…
scMRDR: A scalable and flexible framework for unpaired single-cell multi-omics data integration
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Carnegie Mellon University（卡内基梅隆大学）；The Chinese University of Hong Kong（香港中文大学）；Guangzhou Laboratory（广州实验室）
Stochastic differential equations provide a rich class of flexible generative models, capable of describing a wide range of spatio-temporal processes. A host of recent work looks to learn data-representing SDEs, using neural networks and other flexible function approximators. Despite these advances, learning remains computationally expensive due to the sequential nature of SDE integrators. In this work, we propose an importance-sampling estimator for probabilities of observations of SDEs for the purposes of learning. Crucially, the approach we suggest does not rely on such integrators. The pro…
Cranfield University（克兰菲尔德大学）；University of Warwick（华威大学）；Alan Turing Institute（艾伦·图灵研究所）
Achieving transparency in black-box deep learning algorithms is still an open challenge. High dimensional features and decisions given by deep neural networks (NN) require new algorithms and methods to expose its mechanisms. Current state-of-the-art NN interpretation methods (e.g. Saliency maps, DeepLIFT, LIME, etc.) focus more on the direct relationship between NN outputs and inputs rather than the NN structure and operations itself. In current deep NN operations, there is uncertainty over the exact role played by neurons with fixed activation functions. In this paper, we achieve partially ex…
Quantum Pointwise Convolution: A Flexible and Scalable Approach for Neural Network Enhancement
School of Electrical Engineering, KAIST（韩国科学技术院电气工程学院）；National Synchrotron Radiation Research Center (NSRRC)（国家同步辐射研究中心）；National Center for High-Performance Computing (NCHC)（国家高性能计算中心）
Kernel Neural Operators (KNOs) for Scalable, Memory-efficient, Geometrically-flexible Operator Learning
Kahlert School of Computing（卡勒特计算学院）；University of Utah（犹他大学）；Department of Mathematics（数学系）；Sandia National Laboratories（桑迪亚国家实验室）；Scientific Machine Learning（科学机器学习）；Scientific Computing and Imaging (SCI) Institute（科学计算与成像（SCI）研究所）
Scalable predictive processing framework for multitask caregiving robots
National Institute of Neuroscience, National Center of Neurology and Psychiatry（国立精神·神经医疗研究中心 神经科学研究所）；Future Robotics Organization, Waseda University（早稻田大学未来机器人机构）；Department of Intermedia Art and Science, Waseda University（早稻田大学跨媒体艺术与科学系）
The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；University of Electronic Science and Technology of China（电子科技大学）
Federated learning (FL) is a machine learning field in which researchers try to facilitate model learning process among multiparty without violating privacy protection regulations. Considerable effort has been invested in FL optimization and communication related researches. In this work, we introduce \texttt{FedLab}, a lightweight open-source framework for FL simulation. The design of \texttt{FedLab} focuses on FL algorithm effectiveness and communication efficiency. Also, \texttt{FedLab} is scalable in different deployment scenario. We hope \texttt{FedLab} could provide flexible API as well …
Xilinx Research Labs（赛灵思研究院）；Norwegian University of Science and Technology（挪威科技大学）；University of Sydney（悉尼大学）
Research has shown that convolutional neural networks contain significant redundancy, and high classification accuracy can be obtained even when weights and activations are reduced from floating point to binary values. In this paper, we present FINN, a framework for building fast and flexible FPGA accelerators using a flexible heterogeneous streaming architecture. By utilizing a novel set of optimizations that enable efficient mapping of binarized neural networks to hardware, we implement fully connected, convolutional and pooling layers, with per-layer compute resources being tailored to user…
