稀疏性-存储-精度权衡在简约激活字典学习中的研究
On the Sparsity-Storage-Accuracy Tradeoff in Parsimoniously Activated Dictionary Learning
AI总结:
本文提出简约激活字典学习(PADL)的结构化生成模型解释,推导出稀疏性、存储成本与重建精度之间的权衡关系,并开发出无需手动调参的高效算法,在视觉基准上取得更优性能。
摘要:
字典学习长期以来从优化和概率两个角度被研究。虽然基于元素级稀疏正则化(例如L1稀疏编码)的公式具有完善的概率解释,但许多施加全局约束的结构化变体缺乏清晰且可处理的生成视角。本文重新审视一类在实际中有效但理论上探索不足的字典学习方法,该方法对激活字典原子的数量施加简单的全局正则化,我们称之为简约激活字典学习(PADL)。我们证明PADL在结构化生成模型下等价于最大后验估计,并带有控制全局激活模式的辅助潜变量。该公式使我们能够推导出在原始公式下难以获得的泛化保证。更重要的是,它给出了稀疏性、存储成本和重建精度之间权衡的分析表征,从而能够基于数据估计最优超参数。基于这一联系,我们开发了一种高效且可解释的PADL算法,消除了手动超参数调整,在视觉基准上实现了在可比稀疏性水平下改进的重建性能。我们进一步展示了其在加速视觉-语言模型推理中的实际效用。
英文摘要:
Dictionary learning has long been studied from both optimization and probabilistic perspectives. While formulations with element-wise sparsity regularization (e.g., L1-based sparse coding) admit well-established probabilistic interpretations, many structured variants that impose global constraints lack a clear and tractable generative view. In this paper, we revisit a class of practically effective yet theoretically under-explored dictionary learning methods that impose a simple global regularization on the number of activated dictionary atoms, which we term parsimoniously activated dictionary learning (PADL). We show that PADL admits an equivalent formulation as maximum a posteriori estimation under a structured generative model, with auxiliary latent variables that govern global activation patterns. This formulation allows us to derive generalization guarantees that are difficult to obtain under the original formulation. More importantly, it yields an analytical characterization of the tradeoff between sparsity, storage cost, and reconstruction accuracy, enabling data-driven estimation of optimal hyperparameters. Based on this connection, we develop an efficient and interpretable PADL algorithm that eliminates manual hyperparameter tuning, achieving improved reconstruction performance under comparable sparsity levels on visual benchmarks. We further demonstrate its practical utility in accelerating inference for vision-language models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。