Widely popular transformer-based NLP models such as BERT and Turing-NLG have enormous capacity trending to billions of parameters. Current execution methods demand brute-force resources such as HBM devices and high speed interconnectivity for data parallelism. In this paper, we introduce a new relay-style execution technique called L2L (layer-to-layer) where at any given moment, the device memory is primarily populated only with the executing layer(s)'s footprint. The model resides in the DRAM memory attached to either a CPU or an FPGA as an entity we call eager param-server (EPS). To overcome the bandwidth issues of shuttling parameters to and from EPS, the model is executed a layer at a time across many micro-batches instead of the conventional method of minibatches over whole model. L2L is implemented using 16GB V100 devices for BERT-Large running it with a device batch size of up to 256. Our results show 45% reduction in memory and 40% increase in the throughput compared to the state-of-the-art baseline. L2L is also able to fit models up to 50 Billion parameters on a machine with a single 16GB V100 and 512GB CPU memory and without requiring any model partitioning. L2L scales to arbitrary depth allowing researchers to develop on affordable devices which is a big step toward democratizing AI. By running the optimizer in the host EPS, we show a new form of mixed precision for faster throughput and convergence. In addition, the EPS enables dynamic neural architecture approaches by varying layers across iterations. Finally, we also propose and demonstrate a constant memory variation of L2L and we propose future enhancements. This work has been performed on GPUs first, but also targeted towards all high TFLOPS/Watt accelerators.
Training-Inference Consistent Segmented Execution for Long-Context LLMs
College of Computer Science, Inner Mongolia University, Hohhot 010021, China；National \& Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China；Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China；Thrust of Artificial Intelligence, The Hong Kong University of Science
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））；Createlink Technology（创-link科技）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Institute of Technology（北京理工大学）
