The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models trained on vast amounts of data holds immense promise for practical applications, enhancing industrial productivity and facilitating social development. With the increasing demands on computational capacity, though numerous studies have explored the efficient training, a comprehensive summarization on acceleration techniques of training deep learning models is still much anticipated. In this survey, we present a detailed review for training acceleration. We consider the fundamental update formulation and split its basic components into five main perspectives: (1) data-centric: including dataset regularization, data sampling, and data-centric curriculum learning techniques, which can significantly reduce the computational complexity of the data samples; (2) model-centric, including acceleration of basic modules, compression training, model initialization and model-centric curriculum learning techniques, which focus on accelerating the training via reducing the calculations on parameters; (3) optimization-centric, including the selection of learning rate, the employment of large batchsize, the designs of efficient objectives, and model average techniques, which pay attention to the training policy and improving the generality for the large-scale models; (4) budgeted training, including some distinctive acceleration methods on source-constrained situations; (5) system-centric, including some efficient open-source distributed libraries/systems which provide adequate hardware support for the implementation of acceleration algorithms. By presenting this comprehensive taxonomy, our survey presents a comprehensive review to understand the general mechanisms within each component and their joint interaction.
A Survey and Taxonomy of Distributed Data Mining Research Studies: A Systematic Literature Review
Dian Nuswantoro University（迪安努斯万托罗大学）
内容：数据采掘（Data Mining，DM）方法已经在逐年发展，并且到今天为止，还存在DM技术的增强，其运行速度可以比传统方法快几倍，称为分布式数据采掘（Distributed Data Mining，DDM）。它实际上并不是数据处理中的一个新领域，但近年来这方面的研究受到了越来越多的关注。问题：关于DDM在高声誉期刊和会议上的出版物数量显着增加。这使得研究者很难对DDM有一个全面的了解，需要进一步的研究。解决方案：我们进行了系统的文献回顾，以映射以前在DDM领域的研究。我们的目标是通过识别DDM领域的差距以及热点领域本身来为新的研究提供动力。结果：通过对文献综述中提出的7个研究问题的回答，得出了一些结论。此外，本文还对DDM的研究领域进行了分类。最后，本文对2000年至2015年DDM的发展进行了系统的文献综述，以帮助未来的研究者对DDM的发展现状有一个全面的了解。
LitLLMs, LLMs for Literature Review: Are we there yet?
ServiceNow Research（ServiceNow研究院）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；HEC Montreal（蒙特利尔高等商学院）；University of Waterloo（滑铁卢大学）；University of British Columbia（不列颠哥伦比亚大学）；Polytechnique Montreal（蒙特利尔理工学院）
