FastMPS: Revisit Data Parallel in Large-scale Matrix Product State Sampling
Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);Hefei National Research Center for Physical Sciences at the Microscale(合肥微尺度物质科学国家研究中心);National Supercomputing Center in Wuxi(无锡国家超级计算中心)
Yaojian Chen, Si-Qiu Gong, Lin Gan, Yanfei Liu, An Yang, Yinuo Wang, Chao-yang Lu, Guangwen Yang
Matrix Product State (MPS) is a versatile tensor network representation widely applied in quantum physics, quantum chemistry, and machine learning, etc. MPS sampling serves as a critical fundamental operation in these fields. As the problems become more complex, the scale of MPS is rapidly increasing. Traditional data parallelism is limited by memory and heavy I/O in large-scale MPS. Model parallelism that can handle large-scale MPS imposes rigid process bindings and lacks scalability. This work proposes Fast-MPS, a multi-level parallel framework for scalable MPS sampling. Our design combines data parallelism across samples with tensor parallelism along bond dimensions. We eliminate memory and I/O pressure through compression and overlapping, and revive data parallel in large-scale MPS sampling. We evaluate our approach on Gaussian Boson Sampling, a representative and demanding application. Fast-MPS achieves over 10x speedup compared to existing simulators, scales to thousands of processes, and enables simulations with 8,176 sites and bond dimension chi = 10^4, significantly outperforming the state of the art. Fast-MPS has demonstrated great potential in high-performance tensor network applications.
ParaAnya: Accelerating Parallel Diffusion Sampling with Plug-and-Play Output Caching
National Taiwan University(国立台湾大学);University of Southern California(南加州大学);Artificial Intelligence Center of Research Excellence, National Taiwan University(国立台湾大学卓越人工智能研究中心)
Department of Aerospace Engineering Sciences, University of Colorado Boulder(航空航天工程科学系,科罗拉多大学博尔德分校);Department of Aerospace and Ocean Engineering, Virginia Tech(航空航天与海洋工程系,弗吉尼亚理工学院)
Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity
ICME Stanford University(ICME 斯坦福大学);Department of Mathematics and ICME Stanford University(数学系和ICME 斯坦福大学);Department of Chemistry and ICME Stanford University(化学系和ICME 斯坦福大学)