Identifying words that impact a task's performance more than others is a challenge in natural language processing. Transformers models have recently addressed this issue by incorporating an attention mechanism that assigns greater attention (i.e., relevance) scores to some words than others. Because of the attention mechanism's high computational cost, transformer models usually have an input-length limitation caused by hardware constraints. This limitation applies to many transformers, including the well-known bidirectional encoder representations of the transformer (BERT) model. In this paper, we examined BERT's attention assignment mechanism, focusing on two questions: (1) How can attention be employed to reduce input length? (2) How can attention be used as a control mechanism for conditional text generation? We investigated these questions in the context of a text classification task. We discovered that BERT's early layers assign more critical attention scores for text classification tasks compared to later layers. We demonstrated that the first layer's attention sums could be used to filter tokens in a given sequence, considerably decreasing the input length while maintaining good test accuracy. We also applied filtering, which uses a compute-efficient semantic similarities algorithm, and discovered that retaining approximately 6\% of the original sequence is sufficient to obtain 86.5\% accuracy. Finally, we showed that we could generate data in a stable manner and indistinguishable from the original one by only using a small percentage (10\%) of the tokens with high attention scores according to BERT's first layer.
LaPA$^2$: Length-Aware Prefix and Prompt Attention Augmentation for Long-Form Controllable Text Generation
School of Artificial Intelligence（人工智能学院）；New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）；School of Computing（计算学院）；School of Computer Science and Engineering（计算机科学与工程学院）
Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Facebook AI Research（脸书人工智能研究院）；Allen Institute for AI（艾伦人工智能研究所）
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
MALTA – Machine Learning Theory and Applications Lab, PUCRS – Porto Alegre, Brazil（MALTA机器学习理论与应用实验室，PUCRS-波士顿实验室，巴西）；Kunumi Institute, Brazil（昆米研究所，巴西）
Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
Echotune: A Modular Extractor Leveraging the Variable-Length Nature of Speech in ASR Tasks
Blockchain Industry School, Chengdu University of Information Technology（成都信息工程大学区块链产业学院）；School of Computer Science and Technology, Fudan University（复旦大学计算机科学技术学院）；Department of Computer Science, Victoria University（维多利亚大学计算机科学系）
