The Transformer architecture has shown to be a powerful tool for a wide range of tasks. It is based on the self-attention mechanism, which is an inherently computationally expensive operation with quadratic computational complexity: memory usage and compute time increase quadratically with the length of the input sequences, thus limiting the application of Transformers. In this work, we propose a novel Clustering self-Attention mechanism using Surrogate Tokens (CAST), to optimize the attention computation and achieve efficient transformers. CAST utilizes learnable surrogate tokens to construct a cluster affinity matrix, used to cluster the input sequence and generate novel cluster summaries. The self-attention from within each cluster is then combined with the cluster summaries of other clusters, enabling information flow across the entire input sequence. CAST improves efficiency by reducing the complexity from $O(N^2)$ to $O(αN)$ where N is the sequence length, and α is constant according to the number of clusters and samples per cluster. We show that CAST performs better than or comparable to the baseline Transformers on long-range sequence modeling tasks, while also achieving higher results on time and memory efficiency than other efficient transformers.
Academia Sinica（中央研究院）；University of Waterloo（滑铁卢大学）
This paper presents an empirical study of conversational question reformulation (CQR) with sequence-to-sequence architectures and pretrained language models (PLMs). We leverage PLMs to address the strong token-to-token independence assumption made in the common objective, maximum likelihood estimation, for the CQR task. In CQR benchmarks of task-oriented dialogue systems, we evaluate fine-tuned PLMs on the recently-introduced CANARD dataset as an in-domain task and validate the models using data from the TREC 2019 CAsT Track as an out-domain task. Examining a variety of architectures with diff…
Fudan University（复旦大学）；Shanghai Key Lab of Intelligent Information Processing（上海市智能信息处理重点实验室）；The University of Hong Kong（香港大学）
This paper presents new hierarchically cascaded transformers that can improve data efficiency through attribute surrogates learning and spectral tokens pooling. Vision transformers have recently been thought of as a promising alternative to convolutional neural networks for visual recognition. But when there is no sufficient data, it gets stuck in overfitting and shows inferior performance. To improve data efficiency, we propose hierarchically cascaded transformers that exploit intrinsic image structures through spectral tokens pooling and optimize the learnable parameters through latent attri…
Inpainting physics: self-supervised learning for context-driven fluid simulation
AI for Image-Guided Diagnosis and Therapy, Technical University of Munich（图像引导诊断与治疗人工智能，慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；AI in Healthcare and Medicine, Technical University of Munich（医疗人工智能，慕尼黑技术大学）；Imperial College London（伦敦帝国学院）
Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
School of Informatics, Xiamen University（厦门大学信息学院）；LLM Team, Shopee Pte. Ltd.（Shopee 股份有限公司语言模型团队）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
