arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.18158stat.MLcs.LG
收藏

可分解 Transformer 点过程

Decomposable Transformer Point Processes

University of Cambridge(剑桥大学)

Aristeidis Panos

AI总结:

提出一种可分解 Transformer 点过程框架,通过结合满足马尔可夫性质的对数正态混合模型和 Transformer 架构分别建模事件间时间和标记,规避了计算量大的稀疏算法,在事件预测中达到 SOTA 性能且显著提升推断速度。

摘要:

标记点过程建模的标准范式是通过使用基于注意力(Transformer 风格)的架构对强度函数进行参数化。尽管这些方法具有灵活性,但其推断基于计算量大的稀疏算法。在这项工作中,我们提出了一个框架,它保持了基于注意力架构的优势,并规避了稀疏算法的局限性。该框架依赖于使用满足马尔可夫性质的对数正态混合模型对事件间时间的条件分布进行建模,并使用基于 Transformer 的架构对标记的条件概率质量函数进行建模。所提出的方法在给定序列历史的情况下预测序列的下一个事件方面达到了最先进的性能。实验还揭示了在推断期间不依赖稀疏算法的方法优于依赖该算法的方法的有效性。最后,我们在具有挑战性的长视野预测任务上测试了我们的方法,发现它优于专门为解决该任务而开发的基线;重要的是,与基于稀疏算法的基线相比,推断只需要一小部分时间。

英文摘要:

The standard paradigm of modeling marked point processes is by parameterizing the intensity function using an attention-based (Transformer-style) architecture. Despite the flexibility of these methods, their inference is based on the computationally intensive thinning algorithm. In this work, we propose a framework where the advantages of the attention-based architecture are maintained and the limitation of the thinning algorithm is circumvented. The framework depends on modeling the conditional distribution of inter-event times with a mixture of log-normals satisfying a Markov property and the conditional probability mass function for the marks with a Transformer-based architecture. The proposed method attains state-of-the-art performance in predicting the next event of a sequence given its history. The experiments also reveal the efficacy of the methods that do not rely on the thinning algorithm during inference over the ones they do. Finally, we test our method on the challenging long-horizon prediction task and find that it outperforms a baseline developed specifically for tackling this task; importantly, inference requires just a fraction of time compared to the thinning-based baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑