Recent studies have demonstrated that the performance of transformers on the task of language modeling obeys a power-law relationship with model size over six orders of magnitude. While transformers exhibit impressive scaling, their performance hinges on processing large amounts of data, and their computational and memory requirements grow quadratically with sequence length. Motivated by these considerations, we construct a Legendre Memory Unit based model that introduces a general prior for sequence processing and exhibits an $O(n)$ and $O(n \ln n)$ (or better) dependency for memory and computation respectively. Over three orders of magnitude, we show that our new architecture attains the same accuracy as transformers with 10x fewer tokens. We also show that for the same amount of training our model improves the loss over transformers about as much as transformers improve over LSTMs. Additionally, we demonstrate that adding global self-attention complements our architecture and the augmented model improves performance even further.
MixUp is a computer vision data augmentation technique that uses convex interpolations of input data and their labels to enhance model generalization during training. However, the application of MixUp to the natural language understanding (NLU) domain has been limited, due to the difficulty of interpolating text directly in the input space. In this study, we propose MixUp methods at the Input, Manifold, and sentence embedding levels for the transformer architecture, and apply them to finetune the BERT model for a diverse set of NLU tasks. We find that MixUp can improve model performance, as we…
In this paper, we report improved results of the Fake News Challenge Stage 1 (FNC-1) stance detection task. This gain in performance is due to the generalization power of large language models based on Transformer architecture, invented, trained and publicly released over the last two years. Specifically (1) we improved the FNC-1 best performing model adding BERT sentence embedding of input sequences as a model feature, (2) we fine-tuned BERT, XLNet, and RoBERTa transformers on FNC-1 extended dataset and obtained state-of-the-art results on FNC-1 task.
This paper proposes a Transformer-based model to generate equations for math word problems. It achieves much better results than RNN models when copy and align mechanisms are not used, and can outperform complex copy and align RNN models. We also show that training a Transformer jointly in a generation task with two decoders, left-to-right and right-to-left, is beneficial. Such a Transformer performs better than the one with just one decoder not only because of the ensemble effect, but also because it improves the encoder training procedure. We also experiment with adding reinforcement learnin…
Enhancing Tropical Cyclone Path Forecasting with an Improved Transformer Network
Faculty of Electronics and Telecommunications University of Engineering and Technology, VNU Hanoi, Vietnam（电子电信学院 工程技术大学 巴黎越南河内分校）；Faculty of Information Technology University of Engineering and Technology, VNU Hanoi, Vietnam（信息技术学院 工程技术大学 巴黎越南河内分校）
Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）；Microsoft Research Asia（微软亚洲研究院）；Fudan University（复旦大学）
The identification of active binding drugs for target proteins (termed as drug-target interaction prediction) is the key challenge in virtual screening, which plays an essential role in drug discovery. Although recent deep learning-based approaches achieved better performance than molecular docking, existing models often neglect certain aspects of the intermolecular information, hindering the performance of prediction. We recognize this problem and propose a novel approach named Intermolecular Graph Transformer (IGT) that employs a dedicated attention mechanism to model intermolecular informat…
As opposed to natural languages, source code understanding is influenced by grammatical relationships between tokens regardless of their identifier name. Graph representations of source code such as Abstract Syntax Tree (AST) can capture relationships between tokens that are not obvious from the source code. We propose a novel method, GN-Transformer to learn end-to-end on a fused sequence and graph modality we call Syntax-Code-Graph (SCG). GN-Transformer expands on Graph Networks (GN) framework using a self-attention mechanism. SCG is the result of the early fusion between a source code snippe…
Learning Diverse Features in Vision Transformers for Improved Generalization
Idiap Research Institute（Idiap研究所）；Mila（米拉人工智能研究所）；University of Montreal（蒙特利尔大学）；University of Bucharest（布加勒斯特大学）；Gheorghe Mihoc - Caius Iacob Institute of Mathematical Statistics and Applied Mathematics of the Romanian Academy（罗马尼亚科学院Gheorghe Mihoc-Caius Iacob数理统计与应用数学研究所）
The University of Tokyo（东京大学）；National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）
Transformers have shown improved performance when compared to previous architectures for sequence processing such as RNNs. Despite their sizeable performance gains, as recently suggested, the model is computationally expensive to train and with a high parameter budget. In light of this, we explore parameter-sharing methods in Transformers with a specific focus on generative models. We perform an analysis of different parameter sharing/reduction methods and develop the Subformer. Our model combines sandwich-style parameter sharing, which overcomes naive cross-layer parameter sharing in generati…
Meta AI；The Chinese University of Hong Kong（香港中文大学）
Efficient transformer variants with linear time complexity have been developed to mitigate the quadratic computational overhead of the vanilla transformer. Among them are low-rank projection methods such as Linformer and kernel-based Transformers. Despite their unique merits, they usually suffer from a performance drop comparing with the vanilla transformer on many sequence generation tasks, and often fail to obtain computation gain when the generation is short. We propose MemSizer, an approach towards closing the performance gap while improving the efficiency even with short generation. It pr…
Institute of Computational Perception (CP-JKU)（计算感知研究所（CP-JKU））；LIT Artificial Intelligence Lab（LIT人工智能实验室）；Johannes Kepler University（约翰内斯·开普勒大学）
Standard machine learning models for tagging and classifying acoustic signals cannot handle classes that were not seen during training. Zero-Shot (ZS) learning overcomes this restriction by predicting classes based on adaptable class descriptions. This study sets out to investigate the effectiveness of self-attention-based audio embedding architectures for ZS learning. To this end, we compare the very recent patchout spectrogram transformer with two classic convolutional architectures. We evaluate these three architectures on three tasks and on three different benchmark datasets: general-purpo…
FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model
Shanghai AI Laboratory（上海人工智能实验室）；Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；University of Sydney（悉尼大学）；Tsinghua University（清华大学）
Improved context-sensitive transformer model for inland vessel trajectory prediction
Institute of Ship Technology, Ocean Engineering and Transport Systems (ISMT), University of Duisburg-Essen（杜伊斯堡-埃森大学船舶技术、海洋工程与运输系统研究所（ISMT））；German Federal Waterways Engineering and Research Institute（德国联邦航道工程与研究院）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
Context-Aware Rule Mining Using a Dynamic Transformer-Based Framework
University of Minnesota（明尼苏达大学）；Cornell University（康奈尔大学）；Northeastern University（东北大学）；University of Rochester（罗切斯特大学）；University of Miami（迈阿密大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）
Yonsei University（延世大学）；Deakin University（迪肯大学）；MIT Media Lab（麻省理工学院媒体实验室）；University of Oulu（奥卢大学）
This article seeks for a distributed learning solution for the visual transformer (ViT) architectures. Compared to convolutional neural network (CNN) architectures, ViTs often have larger model sizes, and are computationally expensive, making federated learning (FL) ill-suited. Split learning (SL) can detour this problem by splitting a model and communicating the hidden representations at the split-layer, also known as smashed data. Notwithstanding, the smashed data of ViT are as large as and as similar as the input data, negating the communication efficiency of SL while violating data privacy…
The University of Manchester（曼彻斯特大学）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Recently, Transformer model, which has achieved great success in many artificial intelligence fields, has demonstrated its great potential in modeling graph-structured data. Till now, a great variety of Transformers has been proposed to adapt to the graph-structured data. However, a comprehensive literature review and systematical evaluation of these Transformer variants for graphs are still unavailable. It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. In this survey, we provide a comprehensive rev…
Northeastern University（东北大学）；The Chinese University of Hong Kong（香港中文大学）；NiuTrans Research（小牛翻译研究院）
Improving Transformer efficiency has become increasingly attractive recently. A wide range of methods has been proposed, e.g., pruning, quantization, new architectures and etc. But these methods are either sophisticated in implementation or dependent on hardware. In this paper, we show that the efficiency of Transformer can be improved by combining some simple and hardware-agnostic methods, including tuning hyper-parameters, better design choices and training strategies. On the WMT news translation tasks, we improve the inference efficiency of a strong Transformer system by 3.80X on CPU and 2.…
WiTUnet: A U-Shaped Architecture Integrating CNN and Transformer for Improved Feature Alignment and Local Information Fusion
College of Computer Science and Cyber Security, Chengdu University of Technology（成都理工大学计算机与网络安全学院）；College of Geophysics, Chengdu University of Technology（成都理工大学地球物理学院）；College of Mechanical and Vehicle Engineering, Changchun University（长春大学机械与车辆工程学院）
Improved Prognostic Prediction of Pancreatic Cancer Using Multi-Phase CT by Integrating Neural Distance and Texture-Aware Transformer
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Peking University（北京大学）；Hupan Lab（湖畔实验室）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；South China Hospital, Shenzhen University（深圳大学华南医院）；Tianjin Medical University Cancer Institute and Hospital（天津医科大学肿瘤医院）；Changsha Institute for Computing and Digital Economy（长沙计算与数字经济研究院）；Guangdong Provincial People’s Hospital（广东省人民医院）；Shengjing Hospital（盛京医院）
Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformer-based VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learnin…
This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequence history, allowing them to assign higher probabilities to re-occurring sequential patterns. By applying dynamic evaluation to Transformer-XL models, we improve the state of the art on enwik8 from 0.99 to 0.94 bits/char, text8 from 1.08 to 1.04 bits/char, and WikiText-103 from 1…
University of Michigan（密歇根大学）；Capacloud AI（Capacloud人工智能公司）
Objective:Computer vision-based up-to-date accurate damage classification and localization are of decisive importance for infrastructure monitoring, safety, and the serviceability of civil infrastructure. Current state-of-the-art deep learning (DL)-based damage detection models, however, often lack superior feature extraction capability in complex and noisy environments, limiting the development of accurate and reliable object distinction. Method: To this end, we present DenseSPH-YOLOv5, a real-time DL-based high-performance damage detection model where DenseNet blocks have been integrated wit…
Quantifying Logical Consistency in Transformers via Query-Key Alignment
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；CNRS（法国国家科学研究中心）；Université Paris Cité（巴黎西岱大学）；Artificial Intelligence Research Institute (AIRI)（人工智能研究所（AIRI））
Transformers are widely applied to solve natural language understanding and computer vision tasks. While scaling up these architectures leads to improved performance, it often comes at the expense of much higher computational costs. In order for large-scale models to remain practical in real-world systems, there is a need for reducing their computational overhead. In this work, we present the PatchMerger, a simple module that reduces the number of patches or tokens the network has to process by merging them between two consecutive intermediate layers. We show that the PatchMerger achieves a si…
The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream t…
Google Research（谷歌研究）；Univ. Lyon, INSA-Lyon, CNRS, LIRIS（里昂大学、里昂国立应用科学学院、法国国家科学研究中心、里昂信息科学与应用实验室）
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal trai…
RIMformer: An End-to-End Transformer for FMCW Radar Interference Mitigation
School of Electronic and Information Engineering, Beihang University（北京航空航天大学电子信息工程学院）；Hefei Innovation Research Institute of Beihang University（北京航空航天大学合肥创新研究院）
We consider the problem of learning high-level controls over the global structure of generated sequences, particularly in the context of symbolic music generation with complex language models. In this work, we present the Transformer autoencoder, which aggregates encodings of the input data across time to obtain a global representation of style from a given performance. We show it is possible to combine this global representation with other temporally distributed embeddings, enabling improved control over the separate aspects of performance style and melody. Empirically, we demonstrate the eff…
