S0调优：混合循环-注意模型的零开销适应
S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
AI总结：
S0调优通过零推理开销优化循环层状态矩阵，提升HumanEval性能10.8个百分点，且在多个任务中展示出显著的跨域迁移能力。
摘要：
使用约48个经过验证的人类评估训练解决方案，对每个循环层的单个初始状态矩阵进行调优，零推理开销，比LoRA在HumanEval上提升10.8个百分点（p < 0.001）。该方法通过优化每个循环层的一个状态矩阵，同时冻结所有模型权重。在Qwen3.5-4B（GatedDeltaNet混合）上，S0调优将贪心通过率@1提升23.6±1.7个百分点（10种子）。在FalconH1-7B（Mamba-2混合）上，S0达到71.8%±1.3，LoRA达到71.4%±2.4（3种子），在该样本量下统计上不可区分，且无需权重合并。跨域迁移在MATH-500（+4.8个百分点，p=0.00002，8种子）和GSM8K（+2.8个百分点，p=0.0003，10种子）上显著；文本到SQL基准（Spider）显示无迁移，与轨迹引导机制一致。在纯Transformer（Qwen2.5-3B）上，前缀调优控制会降级性能13.9个百分点。在Qwen3.5上，每步状态偏移变体达到27.1个百分点，超过S0和LoRA，但有每步推理成本。总体而言，结果表明，在验证监督稀缺时，循环状态初始化是混合语言模型的强零推理开销PEFT表面。调优的状态是一个约48MB的文件；任务切换不需要权重合并或模型重新加载。代码和库：https://github.com/jackyoung27/s0-tuning。
英文摘要：
Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval. The method, which we call S0 tuning, optimizes one state matrix per recurrent layer while freezing all model weights. On Qwen3.5-4B (GatedDeltaNet hybrid), S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds). On FalconH1-7B (Mamba-2 hybrid), S0 reaches 71.8% +/- 1.3 and LoRA reaches 71.4% +/- 2.4 (3 seeds), statistically indistinguishable at this sample size while requiring no weight merging. Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K (+2.8 pp, p = 0.0003, 10 seeds); a text-to-SQL benchmark (Spider) shows no transfer, consistent with the trajectory-steering mechanism. A prefix-tuning control on a pure Transformer (Qwen2.5-3B) degrades performance by -13.9 pp under all nine configurations tested. On Qwen3.5, a per-step state-offset variant reaches +27.1 pp, above both S0 and LoRA but with per-step inference cost. Taken together, the results show that recurrent state initialization is a strong zero-inference-overhead PEFT surface for hybrid language models when verified supervision is scarce. The tuned state is a ~48 MB file; task switching requires no weight merging or model reload. Code and library: https://github.com/jackyoung27/s0-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
