Structured pruning is essential for efficient deployment of Large Language Models (LLMs). The varying sensitivity of LLM sub-blocks to pruning necessitates the identification of optimal non-uniformly pruned models. Existing methods evaluate the importance of layers, attention heads, or weight channels in isolation. Such localized focus ignores the complex global structural dependencies that exist across the model. Training-aware structured pruning addresses global dependencies, but its computational cost can be just as expensive as post-pruning training. To alleviate the computational burden of training-aware pruning and capture global structural dependencies, we propose TraceNAS, a training-free Neural Architecture Search (NAS) framework that jointly explores structured pruning of LLM depth and width. TraceNAS identifies pruned models that maintain a high degree of loss landscape alignment with the pretrained model using a scale-invariant zero-shot proxy, effectively selecting models that exhibit maximal performance potential during post-pruning training. TraceNAS is highly efficient, enabling high-fidelity discovery of pruned models on a single GPU in 8.5 hours, yielding a 10$\times$ reduction in GPU-hours compared to training-aware methods. Evaluations on the Llama and Qwen families demonstrate that TraceNAS is competitive with training-aware baselines across commonsense and reasoning benchmarks.
Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models
University of Trieste（的里雅斯特大学）；University of Milano-Bicocca（米兰-比科卡大学）；University of Groningen（格罗宁根大学）；Khoury College of Computer Sciences, Northeastern University（东北大学科里计算机学院）
When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo（宁波数字孪生研究院，东方理工高等研究院，宁波）；Viterbi School of Engineering, University of Southern California（南加州大学维特比工程学院）；The Hong Kong Polytechnic University（香港理工大学）；LMU Munich（慕尼黑大学）；Saarland University（萨尔大学）
MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents
Nanyang Technological University（南洋理工大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Tsinghua University（清华大学）
KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits
Key Lab of HCST (PKU), MOE（HCST关键实验室（北京大学））；SCS, Peking University, Beijing, China（SCS，北京大学，北京，中国）；Hong Kong University of Science（香港科学大学）；East China Normal University, Shanghai, China（华东师范大学，上海，中国）；Department of Computer Science, Tianjin University, Tianjin, China（天津大学计算机科学系，天津，中国）；School of Computing Science, Simon Fraser University, Burnaby, BC, Canada（Simon Fraser大学计算机科学学院，Burnaby，BC，加拿大）；University of Texas at Dallas, USA（德克萨斯大学达拉斯分校，美国）；Beijing Tongming Lake Information Technology Application Innovation Center, China（北京同铭湖信息技术应用创新中心，中国）；Fudan University Institute of Systems for Advanced Computing, China（复旦大学高级计算系统研究所，中国）；Shanghai Institute of Systems for Open Computing, China（上海开放计算系统研究所，中国）
