Dense embedding models are commonly deployed in commercial search engines, wherein all the document vectors are pre-computed, and near-neighbor search (NNS) is performed with the query vector to find relevant documents. However, the bottleneck of indexing a large number of dense vectors and performing an NNS hurts the query time and accuracy of these models. In this paper, we argue that high-dimensional and ultra-sparse embedding is a significantly superior alternative to dense low-dimensional embedding for both query efficiency and accuracy. Extreme sparsity eliminates the need for NNS by replacing them with simple lookups, while its high dimensionality ensures that the embeddings are informative even when sparse. However, learning extremely high dimensional embeddings leads to blow up in the model size. To make the training feasible, we propose a partitioning algorithm that learns such high dimensional embeddings across multiple GPUs without any communication. This is facilitated by our novel asymmetric mixture of Sparse, Orthogonal, Learned and Random (SOLAR) Embeddings. The label vectors are random, sparse, and near-orthogonal by design, while the query vectors are learned and sparse. We theoretically prove that our way of one-sided learning is equivalent to learning both query and label embeddings. With these unique properties, we can successfully train 500K dimensional SOLAR embeddings for the tasks of searching through 1.6M books and multi-label classification on the three largest public datasets. We achieve superior precision and recall compared to the respective state-of-the-art baselines for each of the tasks with up to 10 times faster speed.
Learning Set Functions that are Sparse in Non-Orthogonal Fourier Bases
ETH Zurich(苏黎世联邦理工学院)
机器学习在离散域上的许多应用,例如在推荐系统或拍卖中学习偏好函数,可以简化为估计在傅立叶域中稀疏的集合函数。在这项工作中,我们提出了一个新的家庭学习傅立叶稀疏集函数的算法。它们最多需要$nk - k \log_2 k + k$查询(集合函数求值),在傅立叶系数的温和条件下,其中$n$是基集的大小,$k$是非零傅立叶系数的数目。与其他工作,专注于正交沃尔什-哈达玛变换,我们的新算法与最近推出的非正交傅立叶变换,提供不同的概念傅立叶稀疏。这些问题在建模时自然会出现,例如,形成替代物和补充物的项目集合。我们证明了几个现实世界的应用程序的有效性。
HadamRNN: Binary and Sparse Ternary Orthogonal RNNs
Institut de Mathématiques de Toulouse(图卢兹数学研究所);Institut de Recherche Technologique Saint Exupéry(圣埃克苏佩里技术研究所);Université de Toulouse(图卢兹大学);CNRS(法国国家科学研究中心)
A Decomposition Framework for Certifiably Optimal Orthogonal Sparse PCA
School of Computer Science and Artificial Intelligence(计算机科学与人工智能学院);Aerospace Information Technology University(航空航天信息技术大学);Academy of Mathematics and Systems Science(数学与系统科学学院);Chinese Academy of Sciences(中国科学院)
Sparse-View X-Ray CT Reconstruction Using $\ell_1$ Prior with Learned Transform
Shanghai Jiao Tong University(上海交通大学);University of Michigan - Shanghai Jiao Tong University Joint Institute(上海交通大学密西根学院);The University of Michigan(密歇根大学)
Learned ISTA with Error-based Thresholding for Adaptive Sparse Coding
Institute for Artificial Intelligence, Tsinghua University(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Department of Automation, Tsinghua University(清华大学自动化系)
Neural-POD: A Plug-and-Play Neural Operator Framework for Infinite-Dimensional Functional Nonlinear Proper Orthogonal Decomposition
Department of Mathematics and Statistics(数学与统计学系);Utah State University(犹他州立大学);School of Electrical and Computer Engineering(电气与计算机工程学院);Purdue University(普渡大学);Department of Mathematics(数学系);School of Mechanical Engineering(机械工程学院)
Neural-POD通过在函数空间中学习非线性正交基函数,提供了一种可集成到降阶模型和算子学习框架中的通用表示模块,用于提升AI for science的泛化能力和可解释性。
Approximating the universal thermal climate index using sparse regression with orthogonal polynomials
Department of Knowledge Technologies, Jo z ef Stefan Institute, Jamova 39, 1000 Ljubljana, Slovenia Faculty of Mathematics;Physics, University of Ljubljana, Jadranska ulica 19, 1000 Ljubljana, Slovenia