Unlocking Full Efficiency of Token Filtering in Large Language Model Training
Shanghai University of Finance and Economics(上海财经大学);Hong Kong University of Science and Technology(香港科技大学);University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学)
Di Chai, Pengbo Li, Feiyuan Zhang, Yilun Jin, Han Tian, Kaiqiang Xu, Binhang Yuan, Dian Shen, Junxue Zhang, Kai Chen
Token filtering has been proposed to enhance the utility of large language models (LLMs) by eliminating inconsequential tokens during training. While usingfewer tokens is expected to reduce computational workloads, existing methods have not yet achieved a real-world efficiency boost. This is primarily due to two factors: (1) existing work has inadequate sparsity for speedup, and (2) token filtering operates within a sparsity range that is non-standard in existing machine learning (ML) libraries and thus cannot be efficiently supported. This paper presents Centrifuge, a system that leverages algorithm and system co-design to unleash the full efficiency of token filtering in LLM training. At the algorithm level, Centrifuge filters activations of inconsequential tokens in the attention backward kernel to amplify the sparsity in backward computation. At the system level, Centrifuge proposes an automatic workflow that transforms sparse GEMM into dimension-reduced dense GEMM for optimized efficiency using standard ML libraries. Evaluations on models with various scales--from 1.1B to 40B--demonstrate that Centrifuge reduces backpropagation time by up to 49.9\% and end-to-end training time by up to 34.7\% when filtering 50\% of tokens. Utility assessments indicate that Centrifuge preserves the utility benefits of token filtering and significantly enhances model performance by up to 26.6\% compared to standard training. Centrifuge is designed for seamless integration into existing LLM training frameworks, enabling systems already utilizing token filtering to accelerate training with just one line of code.
Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
University of California, Santa Barbara(加州大学圣芭芭拉分校);University of California, Santa Cruz(加州大学圣克ruz分校);University of California, Los Angeles(加州大学洛杉矶分校);Purdue University(普渡大学);LMSYS Org(LMSYS组织);Microsoft(微软)
RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization
Nilekani Centre at AI4Bharat(AI4Bharat的尼莱卡尼中心);National Institute of Information and Communications Technology(国家信息通信技术研究所);Indian Institute of Technology Madras(印度马德拉斯理工学院);Flipkart;Microsoft(微软);IIIT D&M Kancheepuram;Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Institute for Infocomm Research (IR), A * STAR, Singapore(新加坡资讯通信研究院(A*STAR));Indian Institute of Technology Bombay(印度孟买理工学院)
Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute(杭州高新区(滨江)区块链与数据安全研究院)
Unlocking the Working Memory of Large Language Models for Latent Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria(林茨ELLIS单元和LIT AI实验室,机器学习研究所,约翰·凯撒大学林茨,奥地利);NXAI GmbH, Linz, Austria(NXAI公司,林茨,奥地利)
Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(复旦大学计算机科学学院上海市数据科学重点实验室);ByteDance AI Lab(字节跳动人工智能实验室);East China Normal University(华东师范大学)
Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling
Computer Vision Group, University of Freiburg(弗赖堡大学计算机视觉组);Neuromedical A.I. Lab, Medical Center -- University of Freiburg(弗赖堡大学医学神经AI实验室);Medical Physics, Medical Center -- University of Freiburg(弗赖堡大学医学物理);Visual Geometry Group, University of Oxford(牛津大学视觉几何组)
ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization
The Chinese University of Hong Kong(香港中文大学);Tongyi Lab, Alibaba Group(阿里云实验室);Penn State University(宾夕法尼亚州立大学);The Hong Kong University of Science(香港科学大学)
WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering
Nanjing University of Science and Technology(南京理工大学);Alibaba Group(阿里巴巴集团);Huzhou Normal University(湖州师范学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家实验室,南京大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学);School of Data Science, The Chinese University of Hong Kong, Shenzhen(数据科学学院,香港中文大学(深圳));Department of Computer Science and Engineering, The Chinese University of Hong Kong(计算机科学与工程系,香港中文大学)