We consider representation learning on periodic graphs encoding crystal materials. Different from regular graphs, periodic graphs consist of a minimum unit cell repeating itself on a regular lattice in 3D space. How to effectively encode these periodic structures poses unique challenges not present in regular graph representation learning. In addition to being E(3) invariant, periodic graph representations need to be periodic invariant. That is, the learned representations should be invariant to shifts of cell boundaries as they are artificially imposed. Furthermore, the periodic repeating patterns need to be captured explicitly as lattices of different sizes and orientations may correspond to different materials. In this work, we propose a transformer architecture, known as Matformer, for periodic graph representation learning. Our Matformer is designed to be invariant to periodicity and can capture repeating patterns explicitly. In particular, Matformer encodes periodic patterns by efficient use of geometric distances between the same atoms in neighboring cells. Experimental results on multiple common benchmark datasets show that our Matformer outperforms baseline methods consistently. In addition, our results demonstrate the importance of periodic invariance and explicit repeating pattern encoding for crystal representation learning.
PPTNet: A Hybrid Periodic Pattern-Transformer Architecture for Traffic Flow Prediction and Congestion Identification
National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University(汽车底盘集成与仿生国家级重点实验室,吉林大学);School of Mechanical Engineering, Shandong University(机械工程学院,山东大学);School of Engineering and Physical Sciences, Heriot-Watt University(工程与物理科学学院,赫瑞-瓦特大学)
PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting
School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Shanghai Key Laboratory of Data Science, Fudan University(复旦大学上海市数据科学重点实验室);Xiaohongshu Inc.(小红书公司)
Do Transformers Have the Ability for Periodicity Generalization?
School of Computer Science, Peking University(北京大学计算机科学学院);College of AI, Tsinghua University(清华大学人工智能学院);School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)科学与工程学院)
Space Group Informed Transformer for Crystalline Materials Generation
Beijing National Laboratory for Condensed Matter Physics(北京凝聚态物理国家实验室);School of Physical Sciences, University of Chinese Academy of Sciences, Beijing 100190, China(中国科学院大学物理科学学院,北京100190,中国);Key Laboratory of Material Simulation Methods and Software of Ministry of Education, College of Physics, Jilin University, Changchun 130012, China(教育部材料模拟方法与软件重点实验室,吉林大学物理学院,长春130012,中国);State Key Laboratory of Superhard Materials, College of Physics, Jilin University, Changchun 130012, P. R. China(超硬材料国家重点实验室,吉林大学物理学院,长春130012,中华人民共和国);Songshan Lake Materials Laboratory, Dongguan, Guangdong 523808, China(松山湖材料实验室,东莞,广东523808,中国)
Predicting Many Crystal Properties via an Adaptive Transformer-based Framework
Fudan University(复旦大学);Shanghai Research Center for Quantum Sciences(上海量子科学研究中心);Institute for Nanoelectronic Devices and Quantum Computing, Zhangjiang Fudan International Innovation Center, Fudan University(复旦大学张江国际创新中心纳米电子器件与量子计算研究所);Hefei National Laboratory(合肥国家实验室)
Wyckoff Transformer: Generation of Symmetric Crystals
Institute for Functional Intelligent Materials University of Singapore(功能智能材料研究所新加坡大学);School of Materials Science and Engineering(材料科学与工程学院);Nanyang Technological University(南洋理工大学);Constructor University(Constructor大学);Constructor Knowledge Labs(Constructor知识实验室);HSE University(莫斯科高等经济大学);Institute of Materials Research and Engineering(材料研究与工程研究所)