MoKA：克罗内克适配器的混合
MoKA: Mixture of Kronecker Adapters
AI总结：
MoKA通过将权重更新建模为克罗内克积的混合并引入门控机制，提升了低秩适配器的表达能力，在指令微调和常识推理任务上超越基线，并将可训练参数减少高达27倍。
摘要：
参数高效微调（PEFT）对于降低大型语言模型（LLMs）的计算开销至关重要。低秩系列适配器通常用于在保持LLMs生成能力的同时有效控制参数规模。然而，由于秩约束导致的有限表达能力往往限制了它们在复杂任务上的性能。我们提出了克罗内克适配器的混合（MoKA），这是一种新一代的克罗内克适配器，通过将权重更新建模为克罗内克积的混合来解决这一局限性。我们提出的适配器利用了一种门控机制，该机制衡量每个克罗内克因子的重要性，从而实现更具表达力的适配。此外，MoKA实现了秩灵活性，在参数效率和准确性之间提供了更好的权衡。为确保硬件效率，我们使用标准矩阵运算重新表述了克罗内克计算，从而能够在GPU优化的硬件上无缝部署。我们使用LLaMA2-7B和LLaMA3-8B模型的低比特量化版本，在指令微调和常识推理任务上进行了广泛实验。MoKA不仅优于PEFT基线，还将可训练参数数量减少了高达27倍，在性能和参数效率之间实现了最先进的权衡。
英文摘要：
Parameter-efficient fine-tuning (PEFT) is essential for reducing the computational overhead of large language models (LLMs). Low-rank family adapters are commonly used to control the parameter size efficiently while maintaining the generative power of LLMs. However, their limited expressiveness due to the rank constraint often restricts their performance on complex tasks. We propose Mixture of Kronecker Adapters (MoKA), a new generation of Kronecker adapters that addresses this limitation by modeling weight updates as a mixture of Kronecker products. Our proposed adapter leverages a gating mechanism that measures the importance of each Kronecker factor, enabling more expressive adaptation. Moreover, MoKA enables a rank flexibility that provides a better trade-off between parameter efficiency and accuracy. To ensure hardware efficiency, we reformulate Kronecker computations using standard matrix operations, allowing seamless deployment on GPU-optimized hardware. We conduct extensive experiments on instruction-tuning and commonsense reasoning tasks using low-bit quantized versions of LLaMA2-7B and LLaMA3-8B models. MoKA not only outperforms PEFT baselines, but also reduces the number of trainable parameters up to 27x, achieving state-of-the-art trade-offs between performance and parameter efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
