LLaVA-FA: 通过傅里叶近似压缩大型多模态模型
LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models
AI总结:
LLaVA-FA通过频域联合低秩和量化近似,实现高效压缩大型多模态模型,采用极坐标量化和对角校准方案,提升压缩效果与效率。
摘要:
大型多模态模型(LMMs)在各种视觉-语言任务上取得了显著的性能,但其庞大的计算和内存成本阻碍了其实际部署。现有的压缩方法往往将低秩分解和量化解耦,导致叠加的重建误差,尤其是在具有跨模态冗余的多模态架构中。为了解决这个问题,我们提出了LLaVA-FA,一种新的高效LMM,它在频域中进行联合低秩加量化近似。通过利用傅里叶变换的去相关和共轭对称性质,LLaVA-FA实现了更紧凑且准确的权重表示。此外,我们引入了专为复矩阵设计的极坐标量化方法PolarQuant,以及一个可选的对角校准(ODC)方案,该方案消除了对大规模校准数据的需求。广泛的实验结果表明,我们提出的LLaVA-FA在多个基准上优于现有的高效多模态模型,同时保持了最小的激活参数和低计算成本,验证了其作为压缩LMMs的有效解决方案的有效性。
英文摘要:
Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。