arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00135cs.CV
收藏

LLaVA-FA: 通过傅里叶近似压缩大型多模态模型

LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models

University of Electronic Science and Technology of China(电子科技大学);Chengdu University of Information Technology(成都信息工程大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))

Pengcheng Zheng, Chaoning Zhang, Jiarong Mo, GuoHui Li, Jiaquan Zhang, Jiahao Zhang, Sihan Cao, Sheng Zheng, Caiyan Qin, Guoqing Wang, Yang Yang

AI总结:

LLaVA-FA通过频域联合低秩和量化近似,实现高效压缩大型多模态模型,采用极坐标量化和对角校准方案,提升压缩效果与效率。

摘要:

大型多模态模型(LMMs)在各种视觉-语言任务上取得了显著的性能,但其庞大的计算和内存成本阻碍了其实际部署。现有的压缩方法往往将低秩分解和量化解耦,导致叠加的重建误差,尤其是在具有跨模态冗余的多模态架构中。为了解决这个问题,我们提出了LLaVA-FA,一种新的高效LMM,它在频域中进行联合低秩加量化近似。通过利用傅里叶变换的去相关和共轭对称性质,LLaVA-FA实现了更紧凑且准确的权重表示。此外,我们引入了专为复矩阵设计的极坐标量化方法PolarQuant,以及一个可选的对角校准(ODC)方案,该方案消除了对大规模校准数据的需求。广泛的实验结果表明,我们提出的LLaVA-FA在多个基准上优于现有的高效多模态模型,同时保持了最小的激活参数和低计算成本,验证了其作为压缩LMMs的有效解决方案的有效性。

英文摘要:

Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑