Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. Weight sharing is one promising solution that encourages weight reuse, effectively reducing memory usage with less performance drop. However, current weight sharing techniques primarily focus on small-scale models like BERT and employ coarse-grained sharing rules, e.g., layer-wise. This becomes limiting given the prevalence of LLMs and sharing an entire layer or block obviously diminishes the flexibility of weight sharing. In this paper, we present a perspective on head-wise shareable attention for large language models. We further propose two memory-efficient methods that share parameters across attention heads, with a specific focus on LLMs. Both of them use the same dynamic strategy to select the shared weight matrices. The first method directly reuses the pre-trained weights without retraining, denoted as $\textbf{DirectShare}$. The second method first post-trains with constraint on weight matrix similarity and then shares, denoted as $\textbf{PostShare}$. Experimental results reveal our head-wise shared models still maintain satisfactory capabilities, demonstrating the feasibility of fine-grained weight sharing applied to LLMs.
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);AI Foundation and Algorithm Lab(人工智能基金会与算法实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Artificial Intelligence Research Institute (AIRI)(人工智能研究院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学)
Steering at the Source: Style Modulation Heads for Robust Persona Control
The University of Tokyo, Tokyo, Japan(东京大学);National Institute of Informatics Research(信息处理研究所);Development Center for Large Language Models, Japan(大型语言模型发展中心)
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
University of Science and Technology of China(中国科学技术大学);State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China(中国科学院计算技术研究所过程器重点实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国)
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Graduate School of Data Science, Seoul National University(数据科学研究生院,首尔国立大学);Department of Rural Systems Engineering, Seoul National University(农村系统工程系,首尔国立大学);Department of Aerospace Engineering, Seoul National University(航空航天工程系,首尔国立大学)
Decoupling and Regulating Modality Heads for Fake News Detection under Missing Modality
Soochow University(苏州大学);Hong Kong University of Science and Technology(香港科技大学);Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);The Hong Kong University of Science and Technology(香港科技大学);FiT, Tencent(腾讯FiT);Alibaba Group(阿里巴巴集团);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);School of Education, Zhejiang Normal University(浙江师范大学教育学院)
Towards Better Multi-head Attention via Channel-wise Sample Permutation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
Multi-head Monotonic Chunkwise Attention For Online Speech Recognition
South China University of Technology(华南理工大学);Tencent(腾讯)
LAS(Listen,Attend and Spell)模型的注意力机制需要整个输入序列来计算注意上下文,因此不适合在线语音识别。为了解决这个问题,我们提出了多头单调块式注意(MTH-MoChA),MoChA的改进版本。MTH-MoChA将输入序列分割成小块,并计算块上的多头注意力。我们还探索了有用的训练策略,如LSTM池化,最小世界出错率训练和SpecAugment,以进一步提高MTH-MoChA的性能。在AISHELL-1数据集上的实验表明,该模型和训练策略使MoChA在测试集上的特征出错率从8.96%提高到7.68%.在另一个18000小时的车内语音数据集,MTH-MoChA获得7.28%的CER,这是显着优于一个国家的最先进的混合系统。