Order-agnostic autoregressive distribution (density) estimation (OADE), i.e., autoregressive distribution estimation where the features can occur in an arbitrary order, is a challenging problem in generative machine learning. Prior work on OADE has encoded feature identity by assigning each feature to a distinct fixed position in an input vector. As a result, architectures built for these inputs must strategically mask either the input or model weights to learn the various conditional distributions necessary for inferring the full joint distribution of the dataset in an order-agnostic way. In this paper, we propose an alternative approach for encoding feature identities, where each feature's identity is included alongside its value in the input. This feature identity encoding strategy allows neural architectures designed for sequential data to be applied to the OADE task without modification. As a proof of concept, we show that a Transformer trained on this input (which we refer to as "the DEformer", i.e., the distribution estimating Transformer) can effectively model binarized-MNIST, approaching the performance of fixed-order autoregressive distribution estimating algorithms while still being entirely order-agnostic. Additionally, we find that the DEformer surpasses the performance of recent flow-based architectures when modeling a tabular dataset.
LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions
University of California, Davis(加州大学戴维斯分校);University of California, Irvine(加州大学伊市分校);University of Southern California(南加州大学);University of South Florida(佛罗里达州立大学)
Statistical Agnostic Mapping: a Framework in Neuroimaging based on Concentration Inequalities
University of Granada(格拉纳达大学);University of Cambridge(剑桥大学)
在70年代,统计学的一个新分支出现了,它专注于在模式识别问题中选择一个函数,该函数满足近似质量与其复杂性之间的确定关系。这些数据驱动的方法主要致力于有限的样本量的依赖性估计的问题,并包括所有的经验样本外的推广方法,如交叉验证(CV)的方法。虽然后者是神经成像中的\emph{not designed for testing competing hypothesis or comparing different models},但是在该理论内存在许多理论发展,其可以用于导出体素或多体素水平的统计不可知(非参数)映射(SAM)。此外,SAM可以缓解i)通过CV方法估计实际风险时有限样本量的不稳定性问题,例如较大的误差条,并提供ii)推论统计中用于假设检验的家族误差(FWE)校正p值图的替代方法。从这个意义上说,我们提出了一种新的框架,在神经成像的基础上集中不等式,这导致(i)一个严格的发展模型验证与小样本/尺寸比,和(ii)一个不太保守的程序比FWE的p值校正,以确定大脑的显着性地图的推断,使用小的上限的实际风险。
CausalWrap: Model-Agnostic Causal Constraint Wrappers for Tabular Synthetic Data
Department of Biostatistics, Vanderbilt University Medical Center(生物统计学系,范德比尔特大学医学中心);Department of Biomedical Informatics, Vanderbilt University Medical Center(生物医学信息学系,范德比尔特大学医学中心)