学习K路D维离散码的紧致嵌入表示
Learning K-way D-dimensional Discrete Code For Compact Embedding Representations
摘要：
诸如词嵌入的嵌入方法已经成为包含离散结构的许多应用的支柱。传统的嵌入方法直接将每个符号与连续的嵌入向量相关联，这相当于基于离散符号的“独热”编码来应用线性变换。尽管其简单性，这种方法产生的参数数量与词汇量呈线性增长，并可能导致过拟合。在这项工作中，我们提出了一个更紧凑的K路D维离散编码方案，以取代“独热”编码。在“KD编码”中，每个符号由$D$维代码表示，并且其每个维度具有$K$的基数。最终的符号嵌入向量可以通过组合代码嵌入向量来生成。为了学习语义上有意义的代码，我们推导出一个宽松的离散优化技术的基础上随机梯度下降。通过采用新的编码系统，参数化的效率可以显着提高（从线性到对数），这也可以减轻过拟合问题。在我们的语言建模实验中，嵌入参数的数量可以减少97%，同时实现类似或更好的性能。
英文摘要：
Embedding methods such as word embedding have become pillars for many applications containing discrete structures. Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying linear transformation based on "one-hot" encoding of the discrete symbols. Despite its simplicity, such approach yields number of parameters that grows linearly with the vocabulary size and can lead to overfitting. In this work we propose a much more compact K-way D-dimensional discrete encoding scheme to replace the "one-hot" encoding. In "KD encoding", each symbol is represented by a $D$-dimensional code, and each of its dimension has a cardinality of $K$. The final symbol embedding vector can be generated by composing the code embedding vectors. To learn the semantically meaningful code, we derive a relaxed discrete optimization technique based on stochastic gradient descent. By adopting the new coding system, the efficiency of parameterization can be significantly improved (from linear to logarithmic), and this can also mitigate the over-fitting problem. In our experiments with language modeling, the number of embedding parameters can be reduced by 97\% while achieving similar or better performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
