Attention sinks are tokens or positions that consistently receive disproportionate attention mass, often beyond what their semantic content would suggest. Despite extensive study in large language models, their behavior and functional relevance in text-to-image (T2I) Transformers remain largely unexplored. To fill this gap and better understand how text-to-image Transformers allocate and route attention, we explore this phenomenon of text-to-image transformers across bidirectional diffusion models and causal autoregressive models. Specifically, we explore whether attention sinks exist, where they emerge, and how they functionally influence generation. Through systematic attention analysis and targeted interventions, we identify that attention sinks emerge in both bidirectional diffusion and autoregressive models but exhibit distinct patterns. The autoregressive model concentrates attention on the first and global-scale tokens, whereas the bidirectional models develop sinks near modality boundaries, within text regions, and at spatial corners. Moreover, these patterns vary across generation stages and are localized to a sparse subset of layers and heads. To assess their functional relevance, we perturb sink-related heads and token positions. It turns out that perturbing sink-ranked heads produces larger average changes than non-sink comparisons, while some sink-position perturbations have limited effects on bidirectional model scores. We further introduce register tokens to redistribute attention sinks and find that several configurations preserve or modestly improve generation performance. Overall, attention sinks are architecture-, inference-stage-, and component-dependent phenomena, providing a basis for mechanistic analysis and principled attention design in text-to-image transformers.
DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Lab. of Computer Network and Information Integration, Southeast University（东南大学计算机网络与信息集成重点实验室）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；Institute of Automation, CAS（中国科学院自动化研究所）
ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Sun Yat-sen University（中山大学）；The Chinese University of Hongkong, Shenzhen（香港中文大学（深圳））
Exploring Motion-Language Alignment for Text-driven Motion Generation
Department of Automation, University of Science and Technology of China（中国科学技术大学自动化系）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，北京通用人工智能研究院）
Reconstruct or Generate: Exploring the Spectrum of Generative Modeling for Cardiac MRI
School of Computation, Information and Technology（计算信息学院）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；School of Medicine, Klinikum Rechts der Isar（医学院，莱茵河右岸医院）；Department of Quantitative Biomedicine（定量生物医学系）；University of Zurich（苏黎世大学）；Department of Computing（计算系）；Imperial College London（伦敦帝国理工学院）
HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation
Wuhan University（武汉大学）；The University of Tokyo（东京大学）；Horizon Robotics；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong（香港中文大学）
Exploring Training and Inference Scaling Laws in Generative Retrieval
National University of Singapore（新加坡国立大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）
LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
Arizona State University（亚利桑那州立大学）；Clemson University（克莱姆森大学）；LinkedIn Corporation（领英公司）；Washington University in St. Louis（圣路易斯华盛顿大学）；Morgan Stanley（摩根大通）
Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning
School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；Alibaba Group（阿里巴巴集团）；School of Statistics, East China Normal University（华东师范大学统计学院）
Autoregressive language modeling (ALM) have been successfully used in self-supervised pre-training in Natural language processing (NLP). However, this paradigm has not achieved comparable results with other self-supervised approach in computer vision (e.g., contrastive learning, mask image modeling). In this paper, we try to find the reason why autoregressive modeling does not work well on vision tasks. To tackle this problem, we fully analyze the limitation of visual autoregressive methods and proposed a novel stochastic autoregressive image modeling (named SAIM) by the two simple designs. Fi…
Empirical Analysis of Decoding Biases in Masked Diffusion Models
School of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Institute for AI, Tsinghua University, China（清华大学人工智能研究院计算机科学与技术系）；Alibaba Group, Hangzhou, China（阿里巴巴集团）
King’s College London（伦敦国王学院）；University College London（伦敦大学学院）；Birkbeck College（伯克贝克学院）；King’s College Hospital NHS Foundation Trust（国王学院医院国民保健服务基金会信托）；St George’s University Hospitals NHS Foundation Trust（圣乔治大学医院国民保健服务基金会信托）；Atkinson Morley Regional Neuroscience Centre（阿特金森·莫利地区神经科学中心）；UCL Queen Square Institute of Neurology（伦敦大学学院皇后广场神经病学研究所）
Deep generative models have emerged as promising tools for detecting arbitrary anomalies in data, dispensing with the necessity for manual labelling. Recently, autoregressive transformers have achieved state-of-the-art performance for anomaly detection in medical imaging. Nonetheless, these models still have some intrinsic weaknesses, such as requiring images to be modelled as 1D sequences, the accumulation of errors during the sampling process, and the significant inference times associated with transformers. Denoising diffusion probabilistic models are a class of non-autoregressive generativ…
SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Nanjing University of Science and Technology（南京理工大学）；City University of Hong Kong（香港城市大学）
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
University of Chinese Academy of Sciences（中国科学院大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；National University of Singapore（新加坡国立大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）
GeoToken: Hierarchical Geolocalization of Images via Next Token Prediction
of Computer Science, University of Southern California, Los Angeles, CA, USA；Computer Engineering, University of Southern California, Los Angeles, CA, USA
