Ever since their conception, Transformers have taken over traditional sequence models in many tasks, such as NLP, image classification, and video/audio processing, for their fast training and superior performance. Much of the merit is attributable to positional encoding and multi-head attention. However, Transformers fall short in learning long-range dependencies mainly due to the quadratic complexity scaled with context length, in terms of both time and space. Consequently, over the past five years, a myriad of methods has been proposed to make Transformers more efficient. In this work, we first take a step back, study and compare existing solutions to long-sequence modeling in terms of their pure mathematical formulation. Specifically, we summarize them using a unified template, given their shared nature of token mixing. Through benchmarks, we then demonstrate that long context length does yield better performance, albeit application-dependent, and traditional Transformer models fall short in taking advantage of long-range dependencies. Next, inspired by emerging sparse models of huge capacity, we propose a machine learning system for handling million-scale dependencies. As a proof of concept, we evaluate the performance of one essential component of this system, namely, the distributed multi-head attention. We show that our algorithm can scale up attention computation by almost $40\times$ using four GeForce RTX 4090 GPUs, compared to vanilla multi-head attention mechanism. We believe this study is an instrumental step towards modeling million-scale dependencies.
Deep Multi-View Spatiotemporal Virtual Graph Neural Network for Significant Citywide Ride-hailing Demand Prediction
College of Systems Engineering, National University of Defense Technology(国防科技大学系统工程学院);Bristol Centre for Functional Nanomaterials, University of Bristol(布里斯托大学布里斯托功能纳米材料中心)
From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos
University of Washington(华盛顿大学);Toyota Technological Institute at Chicago(芝加哥丰田技术学院);Harvard University(哈佛大学);Allen Institute for AI(艾伦人工智能研究所);Cornell University(康奈尔大学)
Multi-view Fuzzy Graph Attention Networks for Enhanced Graph Learning
North Carolina State University(北卡罗来纳州立大学);South China University of Technology(华南理工大学);University of Michigan(密歇根大学);Yeshiva University(叶史瓦大学);Guangxi University(广西大学)
Multi-sensor large-scale dataset for multi-view 3D reconstruction
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);Artificial Intelligence Research Institute(人工智能研究院);Moscow Engineering Physics Institute(莫斯科工程物理学院);New York University(纽约大学);Technical University of Munich(慕尼黑工业大学)
Link Both Views: Cross-Image Routing Attention for Dual-Endpoint V2I Beam-Pair Power Prediction
Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University(山东大学联合山东大学-南洋理工大学人工智能研究中心);School of Software, Shandong University(山东大学软件学院);State Key Laboratory of Photonics and Communications, School of Electronics, Peking University(北京大学电子学系光子与通信国家重点实验室)