密度场状态空间模型：Mamba-2中的1比特蒸馏、高效推理与知识组织
Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2
AI总结：
提出DF-SSM框架，将SSM压缩至1比特骨架加int8低秩校正，应用于Mamba-2 1.3B模型，实现9.7倍压缩和21.4倍推理加速，仅需3200万令牌和6小时蒸馏，并发现模型内部知识组织的三个处理阶段。
摘要：
我们提出了密度场状态空间模型（DF-SSM），这是一个将SSM压缩为1比特骨架并带有int8低秩校正的框架。应用于Mamba-2 1.3B模型，我们得到了一个278 MB的模型（比2.7 GB的FP16教师模型小9.7倍），在GPU上推理速度提升21.4倍（batch=1，相对于mamba-ssm参考实现），同时在下游任务性能上保持在BitMamba-2（一个在150B令牌上从头训练的1.58比特模型）的2-4个百分点以内。蒸馏本身仅需3200万令牌和6小时（在单个A100 GPU上），尽管它假设有一个预训练的FP16教师模型。我们开发了一个优化的推理流水线，结合了用于骨架矩阵乘法的cuBLAS INT8张量核心、用于有状态SSM和卷积操作的自定义CUDA内核，以及用于在GPU和CPU上高效部署的AVX-512 CPU后端。除了压缩，我们还研究了所得模型的内部知识组织，发现了三个不同的处理阶段：意图分类（第0-3层，在没有词汇对齐的抽象空间中操作）、知识检索（第25-35层，事实关联定位在一个5层窗口内）和输出格式化（第36-47层，类别结构消失）。通过对19个类别中445个事实提示的系统分析，我们发现早期层分类是句法的（由模板结构驱动）而非语义的，并且尽管事实回忆较弱，模型仍表现出组织良好的知识表示——这表明表示结构可能先于事实强度。
英文摘要：
We present Density Field State Space Models (DF-SSM), a framework for compressing SSMs to a 1-bit scaffold with int8 low-rank correction. Applied to Mamba-2 1.3B, we achieve a 278 MB model (9.7x smaller than the 2.7 GB FP16 teacher) that runs at 21.4x faster inference on GPU (batch=1, relative to the mamba-ssm reference implementation) while maintaining downstream task performance within 2-4 percentage points of BitMamba-2, a 1.58-bit model trained from scratch on 150B tokens. The distillation itself requires only 32M tokens and 6 hours on a single A100 GPU, though it presupposes a pretrained FP16 teacher. We develop an optimized inference pipeline combining cuBLAS INT8 tensor cores for the scaffold matmul, custom CUDA kernels for stateful SSM and convolution operations, and an AVX-512 CPU backend for efficient deployment on both GPU and CPU. Beyond compression, we investigate the internal knowledge organization of the resulting model, discovering three distinct processing phases: intent classification (layers 0-3, operating in an abstract space with no vocabulary alignment), knowledge retrieval (layers 25-35, where factual associations localize to a 5-layer window), and output formatting (layers 36-47, where category structure dissolves). Through systematic analysis of 445 factual prompts across 19 categories, we find that early-layer classification is syntactic (driven by template structure) rather than semantic, and that the model exhibits well-organized knowledge representations despite weak factual recall--suggesting that representational structure may precede factual strength.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
