本文提出 NiNformer 计算块替代标准 ViT 块，通过引入 Network in Network 结构和由 token mixing 生成的动态门控函数来降低计算与数据需求，在图像分类任务上优于基线架构。
摘要：
注意力机制是 transformer 架构的主要组件；它推动了深度学习在跨越许多领域和涵盖多项任务上的重大进展。在计算机视觉中，注意力机制首先被引入到 Vision Transformer ViT 中，随后其应用扩展到了视觉领域的许多任务中，例如分类、分割、目标检测和图像生成。虽然注意力机制具有极强的表达能力和出色的性能，但它也存在缺点，即计算成本高昂，并且需要相当大规模的数据集才能进行有效优化。为了解决这些缺陷，文献中提出了许多设计来减轻计算负担并缓解对数据规模的要求。视觉领域中此类尝试的例子包括 MLP-Mixer、Conv-Mixer、Perceiver-IO，以及许多具有不同优缺点的其他尝试。本文引入了一个新的计算块作为标准 ViT 块的替代方案。新提出的块通过用 Network in Network 结构替换常规的注意力层来降低计算需求，从而通过由 token mixing 过程生成的逐元素门控函数的动态学习，增强了 MLP-Mixer 的静态方法。大量实验表明，在应用于视觉领域图像分类任务的多个数据集上，所提出的设计比基线架构提供了更好的性能。
英文摘要：
The attention mechanism is the primary component of the transformer architecture; it has led to significant advancements in deep learning spanning many domains and covering multiple tasks. In computer vision, the attention mechanism was first incorporated in the Vision Transformer ViT, and then its usage has expanded into many tasks in the vision domain, such as classification, segmentation, object detection, and image generation. While the attention mechanism is very expressive and capable, it comes with the disadvantage of being computationally expensive and requiring datasets of considerable size for effective optimization. To address these shortcomings, many designs have been proposed in the literature to reduce the computational burden and alleviate the data size requirements. Examples of such attempts in the vision domain are the MLP-Mixer, the Conv-Mixer, the Perciver-IO, and many more attempts with different sets of advantages and disadvantages. This paper introduces a new computational block as an alternative to the standard ViT block. The newly proposed block reduces the computational requirements by replacing the normal attention layers with a Network in Network structure, therefore enhancing the static approach of the MLP-Mixer with a dynamic learning of element-wise gating function generated by a token mixing process. Extensive experimentation shows that the proposed design provides better performance than the baseline architectures on multiple datasets applied in the image classification task of the vision domain.
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Microsoft（微软）
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in model size without significant increases in computational cost. To achieve this, MoE models replace the feedforward sub-layer with Mixture-of-Experts sub-layer in transformers and use a gating network to route each token to its assigned experts. Since the common practice for efficient training of such models requires distributing experts and tokens across different machines, this routing strategy often incurs huge cross…
Quantum-Grassmann-Plucker Token Mixing for Deep Learning-Based Post-Disaster Damage Assessment
Centre for Urban Resilience and Analytics (CURA), Georgia Institute of Technology (Georgia Tech)（佐治亚理工学院城市韧性与分析中心（CURA））；University of Exeter（埃克塞特大学）；University College London (UCL)（伦敦大学学院（UCL））
Although certain vision transformer (ViT) and CNN architectures generalize well on vision tasks, it is often impractical to use them on green, edge, or desktop computing due to their computational requirements for training and even testing. We present WaveMix as an alternative neural architecture that uses a multi-scale 2D discrete wavelet transform (DWT) for spatial token mixing. Unlike ViTs, WaveMix neither unrolls the image nor requires self-attention of quadratic complexity. Additionally, DWT introduces another inductive bias -- besides convolutional filtering -- to utilize the 2D structur…
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；PhiGent Robotics（鉴智机器人）
Data mixing strategies (e.g., CutMix) have shown the ability to greatly improve the performance of convolutional neural networks (CNNs). They mix two images as inputs for training and assign them with a mixed label with the same ratio. While they are shown effective for vision transformers (ViTs), we identify a token fluctuation phenomenon that has suppressed the potential of data mixing strategies. We empirically observe that the contributions of input tokens fluctuate as forward propagating, which might induce a different mixing ratio in the output tokens. The training target computed by the…
PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts
Department of Engineering, King’s College London（伦敦国王学院工程系）；Institute for Digital Communication, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃朗根-纽伦堡弗里德里希-亚历山大大学数字通信研究所）
Tencent（腾讯）；Tsinghua University（清华大学）；Peking University（北京大学）；Tencent AI Lab（腾讯人工智能实验室）
Recently, MLP-like vision models have achieved promising performances on mainstream visual recognition tasks. In contrast with vision transformers and CNNs, the success of MLP-like models shows that simple information fusion operations among tokens and channels can yield a good representation power for deep recognition models. However, existing MLP-like models fuse tokens through static fusion operations, lacking adaptability to the contents of the tokens to be mixed. Thus, customary information fusion procedures are not effective enough. To this end, this paper presents an efficient MLP-like …
In this note we examine the autoregressive generalization of the FNet algorithm, in which self-attention layers from the standard Transformer architecture are substituted with a trivial sparse-uniformsampling procedure based on Fourier transforms. Using the Wikitext-103 benchmark, we demonstratethat FNetAR retains state-of-the-art performance (25.8 ppl) on the task of causal language modelingcompared to a Transformer-XL baseline (24.2 ppl) with only half the number self-attention layers,thus providing further evidence for the superfluity of deep neural networks with heavily compoundedattention…
AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference
Peking University（北京大学）；Institute for Artificial Intelligence（人工智能研究院）；School of Integrated Circuits（集成电路学院）；Institute of Electronic Design Automation（电子设计自动化研究所）；Beijing Advanced Innovation Center for Integrated Circuits（北京集成电路高精尖创新中心）
Recently, visual Transformer (ViT) and its following works abandon the convolution and exploit the self-attention operation, attaining a comparable or even higher accuracy than CNNs. More recently, MLP-Mixer abandons both the convolution and the self-attention operation, proposing an architecture containing only MLP layers. To achieve cross-patch communications, it devises an additional token-mixing MLP besides the channel-mixing MLP. It achieves promising results when training on an extremely large-scale dataset. But it cannot achieve as outstanding performance as its CNN and ViT counterparts…
Mixture-of-Experts (MoE), a conditional computation architecture, achieved promising performance by scaling local module (i.e. feed-forward network) of transformer. However, scaling the cross-token module (i.e. self-attention) is challenging due to the unstable training. This work proposes Sparse-MLP, an all-MLP model which applies sparsely-activated MLPs to cross-token modeling. Specifically, in each Sparse block of our all-MLP model, we apply two stages of MoE layers: one with MLP experts mixing information within channels along image patch dimension, the other with MLP experts mixing inform…
Hierarchical Shift Mixing -- Beyond Dense Attention in Transformers
Dept. of Electrical Engineering, Linköping University, Linköping, Sweden（电子工程系，林霍普大学）；RISE Research Institutes of Sweden, Linköping, Sweden（瑞典RISE研究机构）
TM-UNet: Token-Memory Enhanced Sequential Modeling for Efficient Medical Image Segmentation
Dalian University of Technology（大连理工大学）；University of Lincoln（林肯大学）；University of Nottingham（诺丁汉大学）；Waseda University（早稻田大学）；University of Nottingham Ningbo China（宁波诺丁汉大学）；Yale University（耶鲁大学）
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choic…
JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
Carnegie Mellon University（卡内基梅隆大学）；Amazon GenAI（亚马逊生成人工智能）；James Silberrad Brown Center for Artificial Intelligence（詹姆斯·西伯拉德·布朗人工智能中心）；University of Bristol（布里斯托大学）；Stanford University（斯坦福大学）；Northeastern University（东北大学）；New York University（纽约大学）
Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）
Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization
Peter Grünberg Institute, Neuromorphic Software Ecosystems (PGI-15), Jülich Research Centre, Germany（彼得·格林伯格研究所，神经形态软件生态系统（PGI-15），尤利奇研究中心，德国）；Groningen Cognitive Systems and Materials Center (CogniGron), University of Groningen（Groningen认知系统与材料中心（CogniGron）， Groningen大学）；RWTH Aachen University, Aachen, Germany（亚琛RWTH大学，亚琛，德国）；Department of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany（电气工程与信息技术系，亚琛RWTH大学，亚琛，德国）
Polynomial Mixing for Efficient Self-supervised Speech Encoders
Université Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique（巴黎萨克雷大学，法国国家科学研究中心，跨学科数字科学实验室）；Miles team, Université Paris-Dauphine-PSL（Miles团队，巴黎Dauphine-PSL大学）；Laboratoire Informatique d'Avignon, Avignon Université（阿维尼翁信息实验室，阿维尼翁大学）；LIGM, École Nationale des Ponts et Chaussées（LIGM，巴黎国立桥梁与道路学院）
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，新加坡科技研究局）；Shanghai University of Engineering Science, China（上海工程技术大学）
Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
Center of Statistical Research, School of Statistics and Data Science, Southwestern University of Finance and Economics, Chengdu, China.（统计研究中心，统计与数据科学学院，西南财经大学，成都，中国）；Department of Biomedical Engineering, College of Design and Engineering, National University of Singapore, Singapore（生物医学工程系，设计与工程学院，新加坡国立大学，新加坡）
In the past decade, we have witnessed rapid progress in the machine vision backbone. By introducing the inductive bias from the image processing, convolution neural network (CNN) has achieved excellent performance in numerous computer vision tasks and has been established as \emph{de facto} backbone. In recent years, inspired by the great success achieved by Transformer in NLP tasks, vision Transformer models emerge. Using much less inductive bias, they have achieved promising performance in computer vision tasks compared with their CNN counterparts. More recently, researchers investigate usin…
NVIDIA（英伟达）；California Institute of Technology（加州理工学院）；Stanford University（斯坦福大学）
Vision transformers have delivered tremendous success in representation learning. This is primarily due to effective token mixing through self attention. However, this scales quadratically with the number of pixels, which becomes infeasible for high-resolution inputs. To cope with this challenge, we propose Adaptive Fourier Neural Operator (AFNO) as an efficient token mixer that learns to mix in the Fourier domain. AFNO is based on a principled foundation of operator learning which allows us to frame token mixing as a continuous global convolution without any dependence on the input resolution…
The three existing dominant network families, i.e., CNNs, Transformers, and MLPs, differ from each other mainly in the ways of fusing spatial contextual information, leaving designing more effective token-mixing mechanisms at the core of backbone architecture development. In this work, we propose an innovative token-mixer, dubbed Active Token Mixer (ATM), to actively incorporate flexible contextual information distributed across different channels from other tokens into the given query token. This fundamental operator actively predicts where to capture useful contexts and learns how to fuse th…
DVLA-RL++: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning
School of Software, Shandong University（山东大学软件学院）；Shenzhen Loop Area Institute（深圳河套学院）；School of Computer and Artificial Intelligence, Shandong University of Finance and Economics（山东财经大学计算机与人工智能学院）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）
BIG-MoE: Bypass Isolated Gating MoE for Generalized Multimodal Face Anti-Spoofing
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Great Bay University（大湾区大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学大数据系统计算技术国家工程实验室）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）
CHASM: Cross-frequency Harmonized Axis-Separable Mixing for Spectral Token Operators
University of Southampton（南安普顿大学）；University of Queensland（昆士兰大学）；Chengdu University of Technology（成都理工大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Revisiting Global Token Mixing in Task-Dependent MRI Restoration: Insights from Minimal Gated CNN Baselines
Dept. of Radiology & Imaging Sciences, University of Utah, Salt Lake City, UT, USA（放射学与成像科学系，犹他大学，盐湖城，UT，USA）；Dept. of Electrical & Computer Engineering, University of Utah, Salt Lake City, UT, USA（电气与计算机工程系，犹他大学，盐湖城，UT，USA）；Dept. of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE（计算机视觉系，莫扎德人工智能大学，阿布扎赫，EUA）；Dept. of Biomedical Engineering, University of Utah, Salt Lake City, UT, USA（生物医学工程系，犹他大学，盐湖城，UT，USA）；Dept. of Electrical & Computer Engineering, University of Washington, Seattle, WA, USA（电气与计算机工程系，华盛顿大学，西雅图，WA，USA）
The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Microsoft Azure AI（微软Azure人工智能）
Token-mixing multi-layer perceptron (MLP) models have shown competitive performance in computer vision tasks with a simple architecture and relatively small computational cost. Their success in maintaining computation efficiency is mainly attributed to avoiding the use of self-attention that is often computationally heavy, yet this is at the expense of not being able to mix tokens both globally and locally. In this paper, to exploit both global and local dependencies without self-attention, we present Mix-Shift-MLP (MS-MLP) which makes the size of the local receptive field used for mixing incr…
Convolutional Neural Networks (CNNs) have been regarded as the go-to models for visual recognition. More recently, convolution-free networks, based on multi-head self-attention (MSA) or multi-layer perceptrons (MLPs), become more and more popular. Nevertheless, it is not trivial when utilizing these newly-minted networks for video recognition due to the large variations and complexities in video data. In this paper, we present MLP-3D networks, a novel MLP-like 3D architecture for video recognition. Specifically, the architecture consists of MLP-3D blocks, where each block contains one MLP appl…
ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generation
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；University of Electronic Science and Technology of China（电子科技大学）；School of Computer Science, Peking University（北京大学计算机科学学院）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）
DeGMix: Efficient Multi-Task Dense Prediction with Deformable and Gating Mixer
Department of Computer Science & Technology, Institute for AI, Tsinghua University（计算机科学与技术系，人工智能研究院，清华大学）；School of Computer Science, Wuhan University（计算机科学学院，武汉大学）；King Abdullah University of Science and Technology（卡布斯大学）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Pohang University of Science and Technology (POSTECH)（浦项科技大学）
MLP-Mixer has newly appeared as a new challenger against the realm of CNNs and transformer. Despite its simplicity compared to transformer, the concept of channel-mixing MLPs and token-mixing MLPs achieves noticeable performance in visual recognition tasks. Unlike images, point clouds are inherently sparse, unordered and irregular, which limits the direct use of MLP-Mixer for point cloud understanding. In this paper, we propose PointMixer, a universal point set operator that facilitates information sharing among unstructured 3D points. By simply replacing token-mixing MLPs with a softmax funct…
