Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Chinese Academy of Sciences(中国科学院);Kuaishou Technology(快手科技);Tsinghua University(清华大学)
Leiyu Pan, Zhenpeng Su, Minxuan Lv, Yizhe Xiong, Xiangwen Zhang, Zijia Lin, Hui Chen, Jungong Han, Guiguang Ding, Cheng Luo, Di Zhang, Kun Gai, Deyi Xiong
Large language models have demonstrated exceptional performance across a wide range of tasks. However, dense models usually suffer from sparse activation, where many activation values tend towards zero (i.e., being inactivated). We argue that this could restrict the efficient exploration of model representation space. To mitigate this issue, we propose Finedeep, a deep-layered fine-grained expert architecture for dense models. Our framework partitions the feed-forward neural network layers of traditional dense models into small experts, arranges them across multiple sub-layers. A novel routing mechanism is proposed to determine each expert's contribution. We conduct extensive experiments across various model sizes, demonstrating that our approach significantly outperforms traditional dense architectures in terms of perplexity and benchmark performance while maintaining a comparable number of parameters and floating-point operations. Moreover, we find that Finedeep achieves optimal results when balancing depth and width, specifically by adjusting the number of expert sub-layers and the number of experts per sub-layer. Empirical results confirm that Finedeep effectively alleviates sparse activation and efficiently utilizes representation capacity in dense models.
AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);School of Artificial Intelligence, Sun Yat-Sen University(中山大学人工智能学院);Graduate School of Information Science, Hokkaido University(北海道大学信息科学研究院)
LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering
York University(约克大学);Vector Institute(向量研究所);Connected Minds(连接思维公司);Emory University(埃默里大学);Wilfrid Laurier University(威尔弗里德·劳里埃大学);University of Toronto(多伦多大学);University of Guelph(圭尔夫大学);Monark Health(莫纳克健康公司);CIFAR Solution Network(加拿大高级研究院解决方案网络)
Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Models
Provable Responsible AI and Data Analytics (PRADA) Lab(可证明负责任的人工智能和数据分析实验室);King Abdullah University of Science and Technology (KAUST)(卡布斯科学技术大学);University of Copenhagen(哥本哈根大学);University of Macau(澳门大学)
Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models
Human Centric ART, University of Rome Tor Vergata(人类中心ART,罗马大学托尔维加塔分校);ILCC, School of Informatics, University of Edinburgh(ILCC,信息学院,爱丁堡大学);School of Computer Science, University of Sheffield(计算机科学学院,谢菲尔德大学);Almawave S.p.A.(Almawave公司)
MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai AI Lab, Shanghai Innovation Institute(上海人工智能实验室,上海创新研究院)
Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations
City University of Hong Kong, Hong Kong SAR(香港城市大学);Tsinghua University(清华大学);Huawei Noah’s Ark Lab, Hong Kong SAR(华为诺亚实验室(香港));University of Hong Kong(香港大学);Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州))
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
Massachusetts Institute of Technology(麻省理工学院);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);IBM Research(IBM研究院);IST Austria(奥地利科学技术研究所);Neural Magic, Inc.(Neural Magic公司)
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University(自动化系,清华大学);Academy of Medical Engineering and Translational Medicine, Tianjin University(医学工程与转化医学学院,天津大学)
PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding
School of Information Science and Technology, Fudan University(复旦大学信息科学与技术学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)
Mitigating Overthinking in Large Reasoning Models via Manifold Steering
Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);College of AI, Tsinghua University(清华大学人工智能学院);Shanghai Qi Zhi Institute(上海启智研究所);State Key Laboratory of Virtual Reality Technology and Systems, Beihang University(北京航空航天大学虚拟现实技术与系统国家重点实验室)
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
ZIP Lab, Monash University(莫纳什大学ZIP实验室);SenseTime Research(商汤科技研究院);Beihang University(北京航空航天大学);School of Computer and Communication Sciences, EPFL(洛桑联邦理工学院计算机与通信科学学院);University of Science and Technology Beijing(北京科技大学)