DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models
University of Cambridge(剑桥大学);Technical University of Munich(慕尼黑技术大学);University of Oxford(牛津大学);University of Stuttgart(斯图加特大学);LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心)
Zifeng Ding, Yifeng Li, Yuan He, Antonio Norelli, Jingcheng Wu, Volker Tresp, Michael Bronstein, Yunpu Ma
Learning useful representations for continuous-time dynamic graphs (CTDGs) is challenging, due to the concurrent need to span long node interaction histories and grasp nuanced temporal details. In particular, two problems emerge: (1) Encoding longer histories requires more computational resources, making it crucial for CTDG models to maintain low computational complexity to ensure efficiency; (2) Meanwhile, more powerful models are needed to identify and select the most critical temporal information within the extended context provided by longer histories. To address these problems, we propose a CTDG representation learning model named DyGMamba, originating from the popular Mamba state space model (SSM). DyGMamba first leverages a node-level SSM to encode the sequence of historical node interactions. Another time-level SSM is then employed to exploit the temporal patterns hidden in the historical graph, where its output is used to dynamically select the critical information from the interaction history. We validate DyGMamba experimentally on the dynamic link prediction task. The results show that our model achieves state-of-the-art in most cases. DyGMamba also maintains high efficiency in terms of computational resources, making it possible to capture long temporal dependencies with a limited computation budget.
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent
Middle Tennessee State University(中田纳西州立大学);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at the University of California, Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)
Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg(卢森堡大学安全、可靠性与信任跨学科中心);Department of Computer Science, Peking University(北京大学计算机科学系);Institute of Space Internet, Fudan University(复旦大学空间互联网研究院);Department of Electrical and Computer Engineering, The University of British Columbia(不列颠哥伦比亚大学电气与计算机工程系);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);School of Engineering, Edith Cowan University(埃迪斯科文大学工程学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)