发表机构
Qom University of Technology（库姆理工大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
SAPE是一种基于三明治式硬权重共享拓扑的PEFT框架，通过隔离边界变换降低内存与计算开销，在低参数约束下，于RoBERTa-large、LLaMA-3.2等模型的多项任务中取得优于现有方法的性能。
AI 中文摘要
尽管参数高效微调（PEFT）已通过减少可训练参数数量大幅降低了适配大语言模型（LLM）的硬件成本，但近期研究试图通过参数共享进一步改进PEFT。然而，这些方法要么在各层采用均匀参数共享，会延缓收敛；要么依赖动态掩码策略，会增加计算开销。Transformer架构固有的分层结构所启发的共享模式在PEFT中的潜力尚未被探索。为解决这一空白，我们提出SAPE（Sandwich Adapters for Parameter Efficiency，用于参数效率的三明治适配器），这是一种基于三明治式硬权重共享拓扑的PEFT框架。SAPE通过平衡共享组适配器路由中间Transformer层，同时严格隔离输入嵌入和最终投影边界变换以防止梯度干扰。这种设计显著降低了内存消耗，同时消除了与动态参数共享方法相关的计算开销。在仅编码器和因果解码器架构上的广泛评估表明，SAPE在低参数 regime 中达到了最先进的性能。在自然语言理解任务中，SAPE在RoBERTa-large上的表现优于proPETL，仅使用基线10%的参数预算。在使用LLaMA-3.2（3B）进行自然语言生成和世界知识推理，且参数约束严格在~0.6M的情况下，SAPE的表现优于AdaLoRA，在GSM8K上取得了+4.85%的绝对提升，在CommonsenseQA上取得了+3.11%的绝对提升。此外，通过全面的拓扑消融实验，我们明确了固有的容量权衡：虽然硬参数共享强烈正则化语义泛化，但它略微平滑了刚性多步骤算术推理所需的尖锐层间变换。
英文摘要
While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ~0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning.
Comments16 pages, 4 figures, 10 tables, includes appendix
