通过自适应屏蔽Twins层学习有效和高效的嵌入
Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer
摘要：
分类特征的嵌入学习对于基于深度学习的推荐模型（DLRM）至关重要。每个特征值通过嵌入学习过程映射到嵌入向量。传统的方法为来自同一特征字段的所有特征值配置固定且统一的嵌入大小。然而，这样的配置不仅对于嵌入式学习是次优的，而且存储器昂贵。现有的方法，试图解决这些问题，无论是基于规则或神经架构搜索（NAS）的基础上，需要大量的人力设计或网络训练的努力。它们在嵌入尺寸选择或基于热启动的应用中也不灵活。在本文中，我们提出了一种新的和有效的嵌入大小选择方案。具体来说，我们在标准嵌入层后面设计了一个基于自适应掩码双胞胎的层（AMTL）。AMTL生成一个掩码向量来屏蔽每个嵌入向量的不需要的维度。掩码向量带来了选择维度的灵活性，并且可以轻松地将所提出的层添加到未经训练或经过训练的DLRM中。大量的实验评估表明，所提出的计划优于竞争基准上的所有基准任务，也是内存效率，节省60%的内存使用，而不影响任何性能指标。
英文摘要：
Embedding learning for categorical features is crucial for the deep learning-based recommendation models (DLRMs). Each feature value is mapped to an embedding vector via an embedding learning process. Conventional methods configure a fixed and uniform embedding size to all feature values from the same feature field. However, such a configuration is not only sub-optimal for embedding learning but also memory costly. Existing methods that attempt to resolve these problems, either rule-based or neural architecture search (NAS)-based, need extensive efforts on the human design or network training. They are also not flexible in embedding size selection or in warm-start-based applications. In this paper, we propose a novel and effective embedding size selection scheme. Specifically, we design an Adaptively-Masked Twins-based Layer (AMTL) behind the standard embedding layer. AMTL generates a mask vector to mask the undesired dimensions for each embedding vector. The mask vector brings flexibility in selecting the dimensions and the proposed layer can be easily added to either untrained or trained DLRMs. Extensive experimental evaluations show that the proposed scheme outperforms competitive baselines on all the benchmark tasks, and is also memory-efficient, saving 60\% memory usage without compromising any performance metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
