LoRA中的潜在空间分解
Latent Space Factorization in LoRA
AI总结：
提出FVAE-LoRA，通过VAE学习两个因子化潜在空间，分别捕捉任务显著特征与残差信息，在文本、音频、图像任务上优于标准LoRA，并提升分布偏移下的鲁棒性。
摘要：
低秩适配（LoRA）是一种广泛使用的参数高效微调方法。然而，现有的LoRA变体缺乏在学习到的低秩子空间内明确区分任务相关信息的机制，这可能限制下游性能。我们提出了因子化变分自编码器LoRA（FVAE-LoRA），该方法利用变分自编码器（VAE）学习两个不同的潜在空间。我们提出的新颖证据下界（ELBO）公式明确促进了潜在空间之间的因子化，将其中一个潜在空间专门用于任务显著特征，另一个用于残差信息。在文本、音频和图像任务上的大量实验表明，FVAE-LoRA始终优于标准LoRA。此外，虚假相关性评估证实，FVAE-LoRA能更好地隔离任务相关信号，从而在分布偏移下实现更强的鲁棒性。我们的代码公开于：https://github.com/idiap/FVAE-LoRA
英文摘要：
Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning. However, existing LoRA variants lack mechanisms to explicitly disambiguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance. We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA), which leverages a VAE to learn two distinct latent spaces. Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information. Extensive experiments on text, audio, and image tasks demonstrate that FVAE-LoRA consistently outperforms standard LoRA. Moreover, spurious correlation evaluations confirm that FVAE-LoRA better isolates task-relevant signals, leading to improved robustness under distribution shifts. Our code is publicly available at: https://github.com/idiap/FVAE-LoRA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
