Graph Transformers (GTs) have shown strong empirical performance, yet current architectures vary widely in their use of attention mechanisms, positional embeddings (PEs), and expressivity. Existing expressivity results are often tied to specific design choices and lack comprehensive empirical validation on large-scale data. This leaves a gap between theory and practice, preventing generalizable insights that exceed particular application domains. Here, we propose the Generalized-Distance Transformer (GDT), a GT architecture using standard attention that incorporates many advancements for GTs from recent years, and develop a fine-grained understanding of the GDT's representation power in terms of attention and PEs. Through extensive experiments, we identify design choices that consistently perform well across various applications, tasks, and model scales, demonstrating strong performance in a few-shot transfer setting without fine-tuning. Our evaluation covers over eight million graphs with roughly 270M tokens across diverse domains, including image-based object detection, molecular property prediction, code summarization, and out-of-distribution algorithmic reasoning. We distill our theoretical and practical findings into several generalizable insights about effective GT design, training, and inference.
VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Tongji University（同济大学）；ShanghaiTech University（上海科技大学）；Southern University of Science and Technology（南方科技大学）；Jilin University（吉林大学）
Towards Generalizable Deepfake Image Detection with Vision Transformers
Department of Electronics and Communication Engineering（电子与通信工程系）；Department of Information Technology（信息技术系）；Department of Electrical and Electronics Engineering（电气与电子工程系）；National Institute of Technology Karnataka (NITK), Surathkal - 575025, India（卡纳塔克国家理工学院（NITK），苏拉塔卡尔-575025，印度）
GOPT: Generalizable Online 3D Bin Packing via Transformer-based Deep Reinforcement Learning
State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology（华中科技大学机械科学与工程学院智能制造装备与技术国家重点实验室）；BOSCH Corporate Research, China（博世中国研究中心）；State Key Laboratory of High-end Heavy-load Robots, Midea Group（美的集团高端重载机器人国家重点实验室）；Midea Corporate Research Center（美的集团企业研究中心）
Generalizable automated ischaemic stroke lesion segmentation with vision transformers
University College London（伦敦大学学院）；UCL Queen Square Institute of Neurology（伦敦大学学院女王广场神经病学研究所）；National Hospital for Neurology and Neurosurgery（国家神经病学与神经外科医院）；King’s College London（伦敦国王学院）；Institut des Maladies Neurodégénératives（神经退行性疾病研究所）；CNRS（法国国家科学研究中心）；CEA（法国原子能和替代能源委员会）；University of Bordeaux（波尔多大学）
We propose a new model for making generalizable and diverse retrosynthetic reaction predictions. Given a target compound, the task is to predict the likely chemical reactants to produce the target. This generative task can be framed as a sequence-to-sequence problem by using the SMILES representations of the molecules. Building on top of the popular Transformer architecture, we propose two novel pre-training methods that construct relevant auxiliary tasks (plausible reactions) for our problem. Furthermore, we incorporate a discrete latent variable model into the architecture to encourage the m…
Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Mila-Quebec AI Institute（米拉-魁北克人工智能研究所）；Nvidia Research（英伟达研究院）
Human visual perception can easily generalize to out-of-distributed visual data, which is far beyond the capability of modern machine learning models. Domain generalization (DG) aims to close this gap, with existing DG methods mainly focusing on the loss function design. In this paper, we propose to explore an orthogonal direction, i.e., the design of the backbone architecture. It is motivated by an empirical finding that transformer-based models trained with empirical risk minimization (ERM) outperform CNN-based models employing state-of-the-art (SOTA) DG algorithms on multiple DG datasets. W…
In this paper, we theoretically prove that gradient descent can find a global minimum of non-convex optimization of all layers for nonlinear deep neural networks of sizes commonly encountered in practice. The theory developed in this paper only requires the practical degrees of over-parameterization unlike previous theories. Our theory only requires the number of trainable parameters to increase linearly as the number of training samples increases. This allows the size of the deep neural networks to be consistent with practice and to be several orders of magnitude smaller than that required by…
Toyota Research Institute（丰田研究院）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Modern 3D computer vision leverages learning to boost geometric reasoning, mapping image data to classical structures such as cost volumes or epipolar constraints to improve matching. These architectures are specialized according to the particular problem, and thus require significant task-specific tuning, often leading to poor domain generalization performance. Recently, generalist Transformer architectures have achieved impressive results in tasks such as optical flow and depth estimation by encoding geometric priors as inputs rather than as enforced constraints. In this paper, we extend thi…
LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series
McGill University（麦吉尔大学）；Mila - Quebec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；University of Toronto（多伦多大学）；Concordia University（康科迪亚大学）；com（42.com）
Vision transformers (ViT) have recently attracted considerable attentions, but the huge computational cost remains an issue for practical deployment. Previous ViT pruning methods tend to prune the model along one dimension solely, which may suffer from excessive reduction and lead to sub-optimal model quality. In contrast, we advocate a multi-dimensional ViT compression paradigm, and propose to harness the redundancy reduction from attention head, neuron and sequence dimensions jointly. We firstly propose a statistical dependence based pruning criterion that is generalizable to different dimen…
Vision Transformer (ViT) has recently demonstrated promise in computer vision problems. However, unlike Convolutional Neural Networks (CNN), it is known that the performance of ViT saturates quickly with depth increasing, due to the observed attention collapse or patch uniformity. Despite a couple of empirical solutions, a rigorous framework studying on this scalability issue remains elusive. In this paper, we first establish a rigorous theory framework to analyze ViT features from the Fourier spectrum domain. We show that the self-attention mechanism inherently amounts to a low-pass filter, w…
TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search
Department of EECS, University of California, Irvine（电子工程与计算机科学系，加州大学伊市分校）；Department of Computer Science, University of California, San Diego（计算机科学系，加州大学圣地亚哥分校）
Although representational retrieval models based on Transformers have been able to make major advances in the past few years, and despite the widely accepted conventions and best-practices for testing such models, a $\textit{standardized}$ evaluation framework for testing them has not been developed. In this work, we formalize the best practices and conventions followed by researchers in the literature, paving the path for more standardized evaluations - and therefore more fair comparisons between the models. Our framework (1) embeds the documents and queries; (2) for each query-document pair,…
