arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.06989cs.LG
收藏

DEformer:一种估计Transformer的阶数无关分布

The DEformer: An Order-Agnostic Distribution Estimating Transformer

Auburn University(奥本大学)

Michael A. Alcorn, Anh Nguyen

摘要:

阶数不可知的自回归分布(密度)估计(OADE),即,特征可以以任意顺序出现的自回归分布估计是生成式机器学习中的一个具有挑战性的问题。OADE上的先前工作通过将每个特征分配到输入向量中的不同固定位置来编码特征标识。因此,为这些输入构建的架构必须策略性地屏蔽输入或模型权重,以学习以顺序不可知的方式推断数据集的完整联合分布所需的各种条件分布。在本文中,我们提出了一种替代方法编码功能的身份,其中每个功能的身份包括在其输入值旁边。这种特征标识编码策略允许为顺序数据设计的神经架构不经修改地应用于OADE任务。作为概念证明,我们证明了在此输入上训练的Transformer(我们称为“变形器”,即,分布估计Transformer)可以有效地对二值化MNIST进行建模,接近固定阶自回归分布估计算法的性能,同时仍然是完全阶数不可知的。此外,我们发现,DEformer超过最近的基于流的架构的性能时,建模表格数据集。

英文摘要:

Order-agnostic autoregressive distribution (density) estimation (OADE), i.e., autoregressive distribution estimation where the features can occur in an arbitrary order, is a challenging problem in generative machine learning. Prior work on OADE has encoded feature identity by assigning each feature to a distinct fixed position in an input vector. As a result, architectures built for these inputs must strategically mask either the input or model weights to learn the various conditional distributions necessary for inferring the full joint distribution of the dataset in an order-agnostic way. In this paper, we propose an alternative approach for encoding feature identities, where each feature's identity is included alongside its value in the input. This feature identity encoding strategy allows neural architectures designed for sequential data to be applied to the OADE task without modification. As a proof of concept, we show that a Transformer trained on this input (which we refer to as "the DEformer", i.e., the distribution estimating Transformer) can effectively model binarized-MNIST, approaching the performance of fixed-order autoregressive distribution estimating algorithms while still being entirely order-agnostic. Additionally, we find that the DEformer surpasses the performance of recent flow-based architectures when modeling a tabular dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑