核奇异值分解及其在多数据源上的扩展

中文摘要 AI 辅助

核奇异值分解（KSVD）学习关于非对称核矩阵的一对奇异向量，该矩阵可由两个数据源诱导，例如自注意力中的查询和键，或给定矩阵的行和列。在本工作中，我们将KSVD扩展到多个数据源，即eKSVD，它在非对称核上进行联合非线性特征学习。在原始公式中，与每个数据源相关的投影被联合学习以捕获最大信息，同时结合成对耦合。利用拉格朗日及其Karush-Kuhn-Tucker（KKT）条件，对偶中的优化导致KSVD的Lanczos分解定理中移位特征值问题的推广。此外，推导了一个基于协方差的框架，并结合神经网络（NNs）用于显式特征映射，补充了基于核的解释和优化。数值实验验证了我们的eKSVD与基于Mercer核的方法相比在处理多数据源方面的有效性，而我们部署NNs的创新展示了核方法的巨大灵活性。

英文摘要

Kernel Singular Value Decomposition (KSVD) learns a pair of singular vectors w.r.t. an asymmetric kernel matrix, which can be induced by two data sources, e.g., the queries and keys in self-attention or the rows and columns of a given matrix. In this work, we extend KSVD to multiple data sources, namely eKSVD, which conducts joint nonlinear feature learning upon asymmetric kernels. In the primal formulation, the projections associated with each data source are jointly learned to capture maximal information, while incorporating pair-wise couplings. With the Lagrangian and its Karush-Kuhn-Tucker (KKT) conditions, the optimization in the dual leads to a generalization of the shifted eigenvalue problem in Lanczos decomposition theorem of KSVD. Further, a covariance-based framework is derived together with using neural networks (NNs) for explicit feature mappings, complementary to the kernel-based interpretation and optimization. Numerical experiments verify the effectiveness of our eKSVD compared to methods based on Mercer kernels for tackling multiple data sources, and our innovation of deploying NNs demonstrates great flexibility for kernel methods.

发表机构

• KU Leuven（鲁汶大学）

机构由 AI 辅助整理，请以论文原文为准。