AI 中文总结
Prox是一种两阶段无训练的SwiGLU前馈网络激活稀疏化框架，通过近似中间通道显著性构建掩码，在多类大语言模型上实现了优于基线的性能与解码加速，且兼容量化和稀疏注意力。
AI 中文摘要
前馈网络（FFNs）在大语言模型（LLM）推理中主导内存流量与计算，是激活稀疏化的主要目标。但现有无训练方法因通道选择策略的局限，在高稀疏度下会出现显著的模型质量下降。我们观察到SwiGLU中间状态可提供高效的通道选择信号，不过获取该信号需要代价高昂的密集计算。为解决此问题，我们提出Prox，这是一种用于稀疏SwiGLU FFNs的两阶段无训练框架。Prox依赖关键洞察：稀疏执行仅需中间状态诱导的通道掩码，该掩码可通过其元素的幅度排名而非精确值来构建。具体而言，第一阶段利用输入稀疏性与量化代理权重构建共享掩码；第二阶段精确计算选中的通道，实现所有三个投影的稀疏执行。在来自六个模型家族的十个LLM上，Prox在所有稀疏度水平下均优于无训练基线，在FFN稀疏度为70%时实现高达1.99倍的端到端解码加速，且与量化和稀疏注意力兼容。
英文摘要
Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present \emph{Prox}, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a $1.99\times$ end-to-end decoding speedup at 70\% FFN sparsity, and is compatible with quantization and sparse attention.
