arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.08597cs.LGcs.AIcs.CLcs.CV
收藏

超越四元数全连通层:超复数乘法的参数化($1/n$)

Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters

NTU(南洋理工大学);Amazon Web Services AI(亚马逊云科技人工智能部门);Google Research(谷歌研究院);ETH Zürich(苏黎世联邦理工学院);VinAI Research(VinAI研究院);Mila(米拉研究所);Université de Montréal(蒙特利尔大学)

Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, Jie Fu

摘要:

最近的工作已经证明了超复杂空间中表示学习的合理成功。具体来说,“具有四元数的全连接层”(4D超复数),它用四元数的Hamilton乘积代替全连接层中的实值矩阵乘法,两者都可以节省参数,只有1/4的可学习参数,并在各种应用中实现相当的性能。然而,一个关键的警告是,超复杂空间只存在于非常少的预定义维度(4D,8D和16 D)。这限制了利用超复数乘法的模型的灵活性。为此,我们建议参数化超复杂乘法,允许模型从数据中学习乘法规则,无论这些规则是否是预定义的。因此,我们的方法不仅包含Hamilton乘积,而且还学习在任何nD超复数空间上操作,与全连接层对应物相比,使用任意$1/n$可学习参数提供更多的架构灵活性。LSTM和Transformer模型在自然语言推理、机器翻译、文本风格转换和主语动词一致性方面的应用实验证明了所提出方法的架构灵活性和有效性。

英文摘要:

Recent works have demonstrated reasonable success of representation learning in hypercomplex space. Specifically, "fully-connected layers with Quaternions" (4D hypercomplex numbers), which replace real-valued matrix multiplications in fully-connected layers with Hamilton products of Quaternions, both enjoy parameter savings with only 1/4 learnable parameters and achieve comparable performance in various applications. However, one key caveat is that hypercomplex space only exists at very few predefined dimensions (4D, 8D, and 16D). This restricts the flexibility of models that leverage hypercomplex multiplications. To this end, we propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predefined. As a result, our method not only subsumes the Hamilton product, but also learns to operate on any arbitrary nD hypercomplex space, providing more architectural flexibility using arbitrarily $1/n$ learnable parameters compared with the fully-connected layer counterpart. Experiments of applications to the LSTM and Transformer models on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architectural flexibility and effectiveness of the proposed approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑