高斯随机权重平均用于大型语言模型的贝叶斯低秩适配
Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models
AI总结：
提出将低秩适配与高斯随机权重平均结合，实现大型语言模型的近似贝叶斯推断，提升泛化与校准性能，并增强对分布偏移的鲁棒性。
摘要：
微调后的大型语言模型（LLMs）常常存在过度自信和校准不佳的问题，尤其是在小数据集上进行微调时。为解决这些挑战，我们提出了一种将低秩适配（LoRA）与高斯随机权重平均（SWAG）简单结合的方法，从而在LLMs中实现近似贝叶斯推断。通过在多个自然语言处理（NLP）基准上的广泛测试，我们证明了这种直接且计算高效的方法能够提升模型的泛化能力和校准性能，与更复杂、可比的LLMs贝叶斯推断方法相比具有竞争力。我们进一步表明，我们的方法对分布偏移表现出更强的鲁棒性，这体现在其在分布外任务上的改进性能。
英文摘要：
Fine-tuned Large Language Models (LLMs) often suffer from overconfidence and poor calibration, particularly when fine-tuned on small datasets. To address these challenges, we propose a simple combination of Low-Rank Adaptation (LoRA) with Gaussian Stochastic Weight Averaging (SWAG), facilitating approximate Bayesian inference in LLMs. Through extensive testing across several Natural Language Processing (NLP) benchmarks, we demonstrate that our straightforward and computationally efficient approach improves model generalization and calibration competitively with comparable, more sophisticated methods for Bayesian inference in LLMs. We further show that our method exhibits greater robustness against distribution shift, as reflected in its improved performance on out-of-distribution tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
