学习K路D维离散码的紧嵌入表示
Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations
摘要：
传统的嵌入方法直接将每个符号与连续的嵌入向量相关联，这相当于基于离散符号的“独热”编码来应用线性变换。尽管其简单性，这种方法产生的参数的数量与词汇量的大小线性增长，并可能导致过拟合。在这项工作中，我们提出了一个更紧凑的K路D维离散编码方案，以取代“独热”编码。在所提出的“KD编码”中，每个符号由基数为$K$的$D$维码表示，并且通过合成码嵌入向量来生成最终符号嵌入向量。为了端到端学习语义上有意义的代码，我们推导出一种基于随机梯度下降的松弛离散优化方法，该方法可以普遍应用于任何具有嵌入层的可微计算图。在我们对从自然语言处理到图卷积网络的各种应用的实验中，嵌入层的总大小可以减少高达98%，同时实现类似或更好的性能。
英文摘要：
Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying a linear transformation based on a "one-hot" encoding of the discrete symbols. Despite its simplicity, such approach yields the number of parameters that grows linearly with the vocabulary size and can lead to overfitting. In this work, we propose a much more compact K-way D-dimensional discrete encoding scheme to replace the "one-hot" encoding. In the proposed "KD encoding", each symbol is represented by a $D$-dimensional code with a cardinality of $K$, and the final symbol embedding vector is generated by composing the code embedding vectors. To end-to-end learn semantically meaningful codes, we derive a relaxed discrete optimization approach based on stochastic gradient descent, which can be generally applied to any differentiable computational graph with an embedding layer. In our experiments with various applications from natural language processing to graph convolutional networks, the total size of the embedding layer can be reduced up to 98\% while achieving similar or better performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
