Transformer-based models have become the backbone of a wide range of tasks, but their intensive computation and data movement incur substantial energy overhead. Analog compute-in-memory (CIM) offers an energy-efficient alternative, yet its inherent hardware non-idealities perturb stored weights and can significantly degrade model performance. Although existing noise-aware training (NAT) methods improve robustness by injecting noise during training, they largely overlook the critical role of attention mechanisms in Transformer inference.To bridge this gap, we analyze how CIM-induced noise reshapes attention in Transformers and identify two key observations: (1) the divergence between clean and noisy attention maps increases substantially in deeper layers, and (2) conventional NAT can remain overconfident even when its attention routes deviate markedly from the clean reference. Motivated by these findings, we propose ABNAT, an attention-based noise-aware training framework that teaches the noisy student to preserve the clean teacher's headwise attention routes rather than merely reproduce answers, without hardware changes or inference overhead; selective variants further reduce training cost.Extensive experiments across vision, language, and multimodal Transformers show that ABNAT consistently improves robustness over corresponding NAT baselines, improving noise-degraded accuracy by up to 11.28% and reducing attention deviation by up to 85.77%.
RACE-IT: A Reconfigurable Analog Computing Engine for In-Memory Transformer Acceleration
Artificial Intelligence Research Lab (AIRL), Hewlett Packard Labs, USA(人工智能研究实验室(AIRL)、惠普实验室,美国);Technion - Israel Institute of Technology, Haifa, Israel(技术ion-以色列理工学院,海法,以色列)
Analogical Learning for Cross-Scenario Generalization: Framework and Application to Intelligent Localization
College of Information Science and Electronic Engineering(信息科学与电子工程学院);Zhejiang University(浙江大学);Zhejiang Provincial Key Laboratory of Multi-modal Communication Networks and Intelligent Information Processing(浙江省多模态通信网络与智能信息处理重点实验室)
Dayforce Supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), through grant DGECR-2022-04531. H.C.M.(Dayforce);Department of Mathematics and Statistics York University(约克大学数学与统计学系);Serrano Academy(塞拉诺学院)
Accelerating Transformers with Spectrum-Preserving Token Merging
University of Science - VNUHCM(胡志明市国家大学自然科学大学);Max Planck Research School for Intelligent Systems (IMPRS-IS)(马克斯·普朗克智能系统研究院(IMPRS-IS));University of Stuttgart(斯图加特大学);Dublin City University(都柏林城市大学);University of Queensland(昆士兰大学);University of Arkansas(阿肯色大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);UC San Diego(加利福尼亚大学圣地亚哥分校);Oldenburg University(奥尔登堡大学);Stanford University(斯坦福大学)
Hybrid Quantum Vision Transformers for Event Classification in High Energy Physics
University of Florida(佛罗里达大学);Polytechnic University of Catalonia(加泰罗尼亚理工大学);Indian Institute of Technology Bhilai(贝拉理工学院);University of Kansas(堪萨斯大学);University of Alabama(阿拉巴马大学);Software Engineering Institute, Carnegie Mellon University(卡内基梅隆大学软件工程研究所);Technische University of München(慕尼黑工业大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室);NLP Group, DCST, IAI, BNRIST, Tsinghua University(清华大学计算机科学与技术系、人工智能研究院、北京国家信息科学与技术研究中心自然语言处理组);International Innovation Center of Tsinghua University, Shanghai(清华大学上海国际创新中心);Pattern Recognition Center, WeChat AI, Tencent Inc(腾讯微信人工智能模式识别中心);Quan Cheng Laboratory(泉城实验室)
Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers
The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Hong Kong University of Science and Technology(香港科技大学);The Simons Institute for the Theory of Computing at the University of California, Berkeley(加州大学伯克利分校西蒙斯理论计算研究所);Rice University(莱斯大学)
How Expressive are Transformers in Spectral Domain for Graphs?
Indian Institute of Technology, Hyderabad(印度理工学院海得拉巴分校);RWTH Aachen(亚琛工业大学);Zerotha Research(泽罗萨研究院);Cerence GmbH(赛轮思公司);The University of Tokyo(东京大学);IBM Research(IBM研究院)