arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.22065cs.LG
收藏

ABNAT:面向模拟存算一体系统上鲁棒Transformer的基于注意力的噪声感知训练

ABNAT: Attention-Based Noise-Aware Training for Robust Transformers on Analog Compute-in-Memory Systems

Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心)

Wenshuai Yao, Wenyong Zhou, Hanyong Shao, Zhiyuan Ning, Yizhe Chen, Yuannuo Feng, Kechao Tang, Ru Huang

AI总结:

该研究针对模拟存算一体系统中Transformer受硬件噪声影响性能下降的问题,提出ABNAT框架,通过保留教师模型多头注意力路径提升鲁棒性,在多模态Transformer上使噪声退化准确率提升最高11.28%。

摘要:

基于Transformer的模型已成为众多任务的核心,但它们密集的计算与数据移动会产生巨大的能耗开销。模拟存算一体(CIM)提供了一种高能效的替代方案,然而其固有的硬件非理想性会扰动存储的权重,大幅降低模型性能。尽管现有的噪声感知训练(NAT)方法通过在训练过程中注入噪声提升了鲁棒性,但它们在很大程度上忽略了Transformer中注意力机制的关键作用。为弥合这一差距,我们分析了CIM诱导的噪声如何重塑Transformer中的注意力,并确定了两个关键发现:(1)在更深层中,干净与带噪声的注意力图之间的差异显著增大;(2)即使注意力路径与干净参考明显偏离,传统NAT仍可能保持过度自信。基于这些发现,我们提出了ABNAT,这是一种基于注意力的噪声感知训练框架,它教导带噪声的学生模型保留干净教师模型的多头注意力路径,而非仅复制答案,且无需硬件改动或增加推理开销;选择性变体进一步降低了训练成本。在视觉、语言和多模态Transformer上的实验表明,ABNAT相较于对应的NAT基线始终提升了鲁棒性,将噪声退化的准确率提升了最高11.28%,并将注意力偏差降低了最高85.77%。

英文摘要:

Transformer-based models have become the backbone of a wide range of tasks, but their intensive computation and data movement incur substantial energy overhead. Analog compute-in-memory (CIM) offers an energy-efficient alternative, yet its inherent hardware non-idealities perturb stored weights and can significantly degrade model performance. Although existing noise-aware training (NAT) methods improve robustness by injecting noise during training, they largely overlook the critical role of attention mechanisms in Transformer inference.To bridge this gap, we analyze how CIM-induced noise reshapes attention in Transformers and identify two key observations: (1) the divergence between clean and noisy attention maps increases substantially in deeper layers, and (2) conventional NAT can remain overconfident even when its attention routes deviate markedly from the clean reference. Motivated by these findings, we propose ABNAT, an attention-based noise-aware training framework that teaches the noisy student to preserve the clean teacher's headwise attention routes rather than merely reproduce answers, without hardware changes or inference overhead; selective variants further reduce training cost.Extensive experiments across vision, language, and multimodal Transformers show that ABNAT consistently improves robustness over corresponding NAT baselines, improving noise-degraded accuracy by up to 11.28% and reducing attention deviation by up to 85.77%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑