arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11408cs.AIcs.LGcs.SYeess.SY
收藏

GHOST:通过分组隐藏状态输出感知选择与截断揭示Mamba2中的幻影状态

GHOST: Unmasking Phantom States in Mamba2 via Grouped Hidden-state Output-aware Selection & Truncation

Rice University(里士满大学)

Michael Menezes, Anastasios Kyrillidis

AI总结:

GHOST通过分组隐藏状态输出感知选择与截断,实现Mamba2中状态维度的高效剪枝,减少50%的状态维度并保持模型性能

摘要:

尽管Mamba2扩展的状态维度增强了时间建模能力,但其带来了显著的推理开销,在自回归生成过程中会饱和带宽。标准的剪枝方法无法解决这一瓶颈:无结构稀疏性使激活保持密集,基于幅度的选择忽略了运行时动态,而基于梯度的方法则带来了巨大的成本。我们引入GHOST(分组隐藏状态输出感知选择与截断),一种结构化剪枝框架,仅使用前向传递统计信息近似控制理论的平衡截断。通过联合测量可控性和可观测性,GHOST在不需反向传播的情况下与基于梯度的方法具有同等的保真度。作为亮点,在从130M到2.7B参数的模型上,我们的方法实现了50%的状态维度减少,同时在WikiText-2上大约增加1个perplexity点。代码可在https://anonymous.4open.science/r/mamba2_ghost-7BCB/上获取。

英文摘要:

While Mamba2's expanded state dimension enhances temporal modeling, it incurs substantial inference overhead that saturates bandwidth during autoregressive generation. Standard pruning methods fail to address this bottleneck: unstructured sparsity leaves activations dense, magnitude-based selection ignores runtime dynamics, and gradient-based methods impose prohibitive costs. We introduce GHOST (Grouped Hidden-state Output-aware Selection and Truncation), a structured pruning framework that approximates control-theoretic balanced truncation using only forward-pass statistics. By jointly measuring controllability and observability, GHOST rivals the fidelity of gradient-based methods without requiring backpropagation. As a highlight, on models ranging from 130M to 2.7B parameters, our approach achieves a 50\% state-dimension reduction with approximately 1 perplexity point increase on WikiText-2. Code is available at https://anonymous.4open.science/r/mamba2_ghost-7BCB/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑