Retrofitting Temporal Graph Neural Networks with Transformer
School of Computer Science, Wuhan University（武汉大学计算机学院）；Centre for Perceptual and Interactive Intelligence (CPII)（感知与交互智能中心）；ETH Zürich（苏黎世联邦理工学院）；eBay（易贝）；Peking University（北京大学）
Temporal graph neural networks (TGNNs) outperform regular GNNs by incorporating time information into graph-based operations. However, TGNNs adopt specialized models (e.g., TGN, TGAT, and APAN ) and require tailored training frameworks (e.g., TGL and ETC). In this paper, we propose TF-TGN, which uses Transformer decoder as the backbone model for TGNN to enjoy Transformer's codebase for efficient training. In particular, Transformer achieves tremendous success for language modeling, and thus the community developed high-performance kernels (e.g., flash-attention and memory-efficient attention) and efficient distributed training schemes (e.g., PyTorch FSDP, DeepSpeed, and Megatron-LM). We observe that TGNN resembles language modeling, i.e., the message aggregation operation between chronologically occurring nodes and their temporal neighbors in TGNNs can be structured as sequence modeling. Beside this similarity, we also incorporate a series of algorithm designs including suffix infilling, temporal graph attention with self-loop, and causal masking self-attention to make TF-TGN work. During training, existing systems are slow in transforming the graph topology and conducting graph sampling. As such, we propose methods to parallelize the CSR format conversion and graph sampling. We also adapt Transformer codebase to train TF-TGN efficiently with multiple GPUs. We experiment with 9 graphs and compare with 2 state-of-the-art TGNN training frameworks. The results show that TF-TGN can accelerate training by over 2.20 while providing comparable or even superior accuracy to existing SOTA TGNNs. TF-TGN is available at https://github.com/qianghuangwhu/TF-TGN.
TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers
Brown University（布朗大学）；New Jersey Institute of Technology（新泽西理工学院）；Massachusetts Institute of Technology（麻省理工学院）；Pacific Northwest National Laboratory（太平洋西北国家实验室）
A temporal scale transformer framework for precise remaining useful life prediction in fuel cells
University College London（伦敦大学学院）；University of Sheffield（谢菲尔德大学）；University of Liverpool（利物浦大学）；Newcastle University（纽卡斯尔大学）；Iowa State University（爱荷华州立大学）
TimeFormer: Transformer with Attention Modulation Empowered by Temporal Characteristics for Time Series Forecasting
Software College, Northeastern University, Shenyang, China（东北大学软件学院）；School of Software, University of Science and Technology of China（中国科学技术大学软件学院）；Xichang Satellite Launch Center, Xichang, China（西昌卫星发射中心）
Hyper-STTN: Hypergraph Augmented Spatial-Temporal Transformer Network for Trajectory Prediction
School of Applied and Creative Computing, Purdue University（普渡大学应用与创意计算学院）；Department of Computer Science and Department of Intelligent Systems Engineering, Indiana University Bloomington（印第安纳大学布卢明顿分校计算机科学系和智能系统工程系）；Department of Applied Artificial Intelligence, Sungkyunkwan University（成均馆大学应用人工智能系）
Diagnostic Spatio-temporal Transformer with Faithful Encoding
Ikerlan Technology Research Centre（伊克兰技术研究中心）；Basque Research and Technology Alliance (BRTA)（巴斯克研究与技术联盟（BRTA））；Mondragon University（蒙德拉贡大学）；IBM Research（IBM研究院）
SiamixFormer: a fully-transformer Siamese network with temporal Fusion for accurate building detection and change detection in bi-temporal remote sensing images
Spatio-Temporal Multi-Task Learning Transformer for Joint Moving Object Detection and Segmentation
Valeo R&D（法雷奥研发中心）
移动物体对于自动驾驶任务具有特殊的重要性。检测移动对象可以通过分割对象像素而被定位为移动对象分割，或者通过生成移动目标的限界框而被定位为移动目标检测。在本文中，我们提出了一个多任务学习架构，基于Transformers，通过一个网络来共同执行这两个任务。由于运动特征对任务的重要性，整个设置基于时空聚合。我们评估的性能与MTL设置，无论是早期共享的编码器，和后期共享的编码器-解码器Transformers器的单独任务架构。对于后者，我们提出了一种新的联合任务查询解码器Transformer，使我们能够有任务专用头的共享模型。为了评估我们的方法，我们使用KITTI MOD [29]数据集。结果显示，在单个任务网络上，移动目标检测的mAP提高了1.5%，移动对象分割的IoU提高了2%。
Spatio-Temporal Transformers for Long-Term NDVI Forecasting
Dept. of Computer Science, Bar-Ilan University（巴伊兰大学计算机科学系）；Dept. of Computer Science, College of Law（法律学院计算机科学系）；Faculty of Civil and Environmental Engineering, Technion Israel Institute of Technology（技术学院环境与土木工程学院）
SSTAF: Spatial-Spectral-Temporal Attention Fusion Transformer for Motor Imagery Classification
Department of Electrical and Electronic Engineering, BRAC University（电子与电气工程系，布拉克大学）；Department of Electrical and Computer Engineering, University of Maryland College Park（电气与计算机工程系，马里兰大学学院公园分校）
Static and multivariate-temporal attentive fusion transformer for readmission risk prediction
School of Computer and Artificial Intelligence, Zhengzhou University（郑州大学计算机与人工智能学院）；Cooperative Innovation Center of Internet Healthcare, Zhengzhou University（郑州大学互联网医疗协同创新中心）；Fuwai Hospital, National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院国家心血管病中心阜外医院）
Residual-based Attention Physics-informed Neural Networks for Spatio-Temporal Ageing Assessment of Transformers Operated in Renewable Power Plants
Mondragon University（蒙dragon大学）；University of the Basque Country (UPV/EHU)（巴斯克大学）；Basque Centre for Applied Mathematics（巴斯克应用数学中心）；KTH, Royal Institute of Technology（皇家理工学院）；Ikerbasque, Basque Foundation for Science（巴斯克科学基金会）
