Code datasets, often collected from diverse and uncontrolled sources such as GitHub, potentially suffer from quality issues, thereby affecting the performance and training efficiency of Large Language Models (LLMs) optimized for code generation. Previous studies demonstrated the benefit of using embedding spaces for data pruning, but they mainly focused on duplicate removal or increasing variety, and in other modalities, such as images. Our work focuses on using embeddings to identify and remove "low-quality" code data. First, we explore features of "low-quality" code in embedding space, through the use of synthetic corruptions. Armed with this knowledge, we devise novel pruning metrics that operate in embedding space to identify and remove low-quality entries in the Stack dataset. We demonstrate the benefits of this synthetic corruption informed pruning (SCIP) approach on the well-established HumanEval and MBPP benchmarks, outperforming existing embedding-based methods. Importantly, we achieve up to a 3% performance improvement over no pruning, thereby showing the promise of insights from synthetic corruptions for data pruning.
Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning
Institute of Statistics and Big Data, Renmin University of China（中国人民大学统计与大数据研究院）；Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China（中国人民大学大数据与国家治理研究院）；Center for Applied Statistics, Institute of Statistics and Big Data, Renmin University of China（中国人民大学统计与大数据研究院应用统计中心）
Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
School of information（信息学院）；Electronics, Beijing Institute of Technology, Beijing（电子学院，北京理工大学，北京）；Department of Computer Science, City University of Hong Kong, Hong Kong（计算机科学系，香港城市大学，香港）；School of Computer Science, Wuhan University, Wuhan（计算机科学学院，武汉大学，武汉）；School of Economics（经济学院）；Management, Communication University of China, Beijing（管理学院，中国传媒大学，北京）；School of Science（科学学院）；Engineering, The Chinese University of Hong Kong (Shenzhen), Shenzhen（工程学院，香港中文大学（深圳），深圳）
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）
HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models
Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Tokyo, Tokyo, Japan（东京大学）；Tokyo Institute of Technology, Tokyo, Japan（东京技术大学）
$ϕ$-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time Exploration and Exploitation
Shanghai AI Lab（上海人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）；National University of Singapore（新加坡国立大学）
RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding
National Center for Mathematics and Interdisciplinary Sciences (NCMIS), AMSS, CAS（中国科学院数学与系统科学研究院国家数学与交叉科学中心）；The University of Tokyo（东京大学）；SKLMS and AMSS, Chinese Academy of Sciences（中国科学院数学与系统科学研究院，科学与工程计算国家重点实验室）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）；Lenovo AI Technology Center（联想AI技术中心）
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Drexel University（德雷塞尔大学）；MIT（麻省理工学院）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Duke University（杜克大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；Center for AI Safety（人工智能安全中心）；University of California, Berkeley（加州大学伯克利分校）；University of Chicago（芝加哥大学）
NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies
Institute for Artificial Intelligence, Peking University, Beijing, China（北京大学人工智能研究院）；Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education（教育部高可信软件技术重点实验室）；School of Computer Science, Peking University, Beijing, China（北京大学计算机学院）；State Key Laboratory of Networking（网络与交换技术国家重点实验室）；School of Computer Science, Beijing University of Posts（北京邮电大学计算机学院）
$\mathcal{V}isi\mathcal{P}runer$: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, EIT, Ningbo（宁波空间智能与数字衍生关键实验室，数字孪生研究院，EIT，宁波）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong Polytechnic University（香港理工大学）；Meituan Inc.（美团公司）；National University of Singapore（新加坡国立大学）
Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机学院）；Cognitive AI Lab, Shanghai Huawei Technologies, China（上海华为技术有限公司认知人工智能实验室）
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China（东北大学计算机科学与工程学院，沈阳 110819，中国）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China（东北大学计算机与通信工程学院，秦皇岛 066004，中国）
Speeding Up Neural Machine Translation Decoding by Cube Pruning
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Oregon State University（俄勒冈州立大学）；Baidu Research（百度研究院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Graph-to-Text Generation with Dynamic Structure Pruning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
