In the rapidly evolving field of deep learning, the demand for models that are both expressive and computationally efficient has never been more critical. This paper introduces Orchid, a novel architecture designed to address the quadratic complexity of traditional attention mechanisms without compromising the ability to capture long-range dependencies and in-context learning. At the core of this architecture lies a new data-dependent global convolution layer, which contextually adapts its kernel conditioned on input sequence using a dedicated conditioning neural network. We design two simple conditioning networks that maintain shift equivariance in our data-dependent convolution operation. The dynamic nature of the proposed convolution kernel grants Orchid high expressivity while maintaining quasilinear scalability for long sequences. We evaluate the proposed model across multiple domains, including language modeling and image classification, to highlight its performance and generality. Our experiments demonstrate that this architecture not only outperforms traditional attention-based architectures such as BERT and Vision Transformers with smaller model sizes, but also extends the feasible sequence length beyond the limitations of the dense attention layers. This achievement represents a significant step towards more efficient and scalable deep learning models for sequence modeling. The code is available at https://github.com/Karami-m/orchid.
Vision Transformers versus convolutional neural networks for fine-grained orchid genus identification in a species-rich, data-poor flora: a controlled benchmark on the Orchidaceae of New Guinea
James Cook University（詹姆斯库克大学）；Southwest Papua Natural Resources Conservation Agency, Ministry of Forestry, Indonesia（印度尼西亚林业部西南巴布亚自然资源保护局）；National Research and Innovation Agency (BRIN), Republic of Indonesia（印度尼西亚共和国国家研究与创新局（BRIN））；Royal Botanic Gardens, Kew（英国皇家植物园邱园）；Australian Tropical Herbarium（澳大利亚热带植物标本馆）；Commonwealth Industrial and Scientific Research Organisation (CSIRO)（澳大利亚联邦科学与工业研究组织（CSIRO））；Australian National Herbarium（澳大利亚国家植物标本馆）
Quantum Pointwise Convolution: A Flexible and Scalable Approach for Neural Network Enhancement
School of Electrical Engineering, KAIST（韩国科学技术院电气工程学院）；National Synchrotron Radiation Research Center (NSRRC)（国家同步辐射研究中心）；National Center for High-Performance Computing (NCHC)（国家高性能计算中心）
Flexible Coded Distributed Convolution Computing for Enhanced Straggler Resilience and Numerical Stability in Distributed CNNs
College of Computer Science, Chongqing University（重庆大学计算机学院）；School of Electrical Engineering, Korea University（韩国大学电气工程学院）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；City University of Hong Kong（香港城市大学）
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Institute for Computational & Mathematical Engineering, Stanford University（斯坦福大学计算与数学工程研究所）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）
