arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.16914stat.MEcs.LGstat.ML
收藏

从统计学角度看待小纵向队列数据中的Transformer

A statistical perspective on transformers for small longitudinal cohort data

Institute of Medical Biometry and Statistics (IMBI), University of Freiburg(弗赖堡大学医学生物统计学研究所);Freiburg Center for Data Analysis, Modeling and AI, University of Freiburg(弗赖堡数据科学与人工智能中心);Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, University Hospital Frankfurt(法兰克福大学精神病学、心身医学与心理治疗系);Department of Neuropsychology and Psychological Resilience Research, Central Institute of Mental Health (ZI), Mannheim, Germany(曼海姆中央心理健康研究所神经心理学与心理韧性研究部);Leibniz Institute for Resilience Research (LIR), Mainz, Germany(莱比锡韧性研究研究所);Department of Psychiatry and Psychotherapy, Johannes Gutenberg University Medical Center Mainz(法兰克福大学医学中心心理医学与心理治疗系);Department of Psychiatry, Psychotherapy and Psychosomatic Medicine, University Medicine Halle, Martin-Luther University Halle-Wittenberg(哈雷-韦茨拉尔大学医学院精神病学、心理治疗与心身医学系);German Center for Mental Health (DZPG), partner site Halle-Jena-Magdeburg(德国心理健康研究中心(DZPG)哈雷-耶纳-马格德堡合作伙伴站点);Neuroimaging Center (NIC), Focus Program Translational Neuroscience (FTN), Johannes Gutenberg University Medical Center, Mainz, Germany(神经成像中心(NIC)、转化神经科学焦点计划(FTN)、法兰克福大学医学中心,马尔堡,德国)
Kiana Farhadyar, Maren Hackenberg, Kira Ahrens, Charlotte Schenk, Bianca Kollmann, Oliver Tüscher, Klaus Lieb, Michael M. Plichta, Andreas Reif, Raffael Kalisch… 展开作者

Kiana Farhadyar, Maren Hackenberg, Kira Ahrens, Charlotte Schenk, Bianca Kollmann, Oliver Tüscher, Klaus Lieb, Michael M. Plichta, Andreas Reif, Raffael Kalisch, Martin Wolkewitz, Moritz Hess, Harald Binder

AI总结:

本文提出了一种简化版的transformer架构,用于处理小纵向队列数据,通过统计学视角结合自回归模型和注意力机制,以识别上下文模式并提升预测性能。

摘要:

纵向队列数据建模通常涉及多个变量之间的复杂时间依赖性。在此情况下,transformer架构,尽管在语言和视觉应用中取得了巨大成功,允许我们考虑到个体历史中最近观察的时间点可能并不总是对立即未来最重要的事实。这是通过根据其值的变换对个体的观察分配注意力权重来实现的。这些想法尚未完全应用于纵向队列数据的一个原因是通常需要大数据集。因此,我们提出了一种简化版的transformer架构,保留了核心的注意力机制,同时减少了需要估计的参数数量,使其更适合具有少量时间点的小数据集。受transformer的统计学视角引导,我们以自回归模型为起点,并将注意力作为基于核的操作与时间衰减相结合,其中多个transformer头的聚合,即不同的候选加权方案,被表达为在不同类型的个体基本特征上积累证据。这还使基于排列的统计检验程序能够识别上下文模式。在模拟研究中,该方法被证明即使在少量个体和时间点的情况下也能恢复上下文依赖性。在一项关于韧性研究的数据应用中,我们识别了压力和心理健康动态的时间模式。这表明,经过适当调整的transformer不仅能实现具有竞争力的预测性能,还能在小数据设置中揭示复杂的上下文依赖性。

英文摘要:

Modeling of longitudinal cohort data typically involves complex temporal dependencies between multiple variables. There, the transformer architecture, which has been highly successful in language and vision applications, allows us to account for the fact that the most recently observed time points in an individual's history may not always be the most important for the immediate future. This is achieved by assigning attention weights to observations of an individual based on a transformation of their values. One reason why these ideas have not yet been fully leveraged for longitudinal cohort data is that typically, large datasets are required. Therefore, we present a simplified transformer architecture that retains the core attention mechanism while reducing the number of parameters to be estimated, to be more suitable for small datasets with few time points. Guided by a statistical perspective on transformers, we use an autoregressive model as a starting point and incorporate attention as a kernel-based operation with temporal decay, where aggregation of multiple transformer heads, i.e. different candidate weighting schemes, is expressed as accumulating evidence on different types of underlying characteristics of individuals. This also enables a permutation-based statistical testing procedure for identifying contextual patterns. In a simulation study, the approach is shown to recover contextual dependencies even with a small number of individuals and time points. In an application to data from a resilience study, we identify temporal patterns in the dynamics of stress and mental health. This indicates that properly adapted transformers can not only achieve competitive predictive performance, but also uncover complex context dependencies in small data settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑