Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu… 展开作者
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, Hao Wu
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for representatio of special values, E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions. Our study covers the main modern neural network architectures - CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters unchanged from the 16-bit baseline training sessions. Our training experiments include large, up to 175B parameter, language models. We also examine FP8 post-training-quantization of language models trained using 16-bit formats that resisted fixed point int8 quantization.
RadField3D: A Data Generator and Data Format for Deep Learning in Radiation-Protection Dosimetry for Medical Applications
Physikalisch-Technische Bundesanstalt (PTB)(物理技术联邦机构);Belgian Nuclear Research Centre (SCK CEN)(比利时核研究中心);Institute for Computer Graphics, Technische Universität Braunschweig(计算机图形学研究所,汉堡工业大学);Physics and Astronomy, University of New Mexico(新墨西哥大学物理与天文学系);Cluster of Excellence PhoenixD, Leibniz University Hannover(卓越中心PhoenixD,莱比锡大学汉诺威分校)
Towards Resource-Efficient Streaming of Large-Scale Medical Image Datasets for Deep Learning
University of Maryland School of Medicine(马里兰大学医学院);University of Maryland Institute for Health Computing(马里兰大学健康计算研究所);St. Jude Children’s Research Hospital(圣犹日儿童研究医院);UTHealth Houston(德克萨斯大学健康科学中心休斯顿分校);Johns Hopkins University School of Medicine(约翰霍普金斯大学医学院)