Diversified Batch Selection for Training Acceleration
Cooperative Medianet Innovation Center（合作 medianet 创新中心）；Shanghai Jiao Tong University（上海交通大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；IHPC, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局信息通信研究所）；Shanghai AI Laboratory（上海人工智能实验室）；College of Computing and Data Science, NTU（南洋理工大学计算与数据科学学院）
Feng Hong, Yueming Lyu, Jiangchao Yao, Ya Zhang, Ivor W. Tsang, Yanfeng Wang
The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption. To save cost, a prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. Although recent efforts achieve advancements by measuring the impact of each sample on generalization, their reliance on additional reference models inherently limits their practical applications, when there are no such ideal models available. On the other hand, the vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner, which sacrifices the diversity and induces the redundancy. To tackle this dilemma, we propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples. Specifically, we define a novel selection objective that measures the group-wise orthogonalized representativeness to combat the redundancy issue of previous sample-wise criteria, and provide a principled selection-efficient realization. Extensive experiments across various tasks demonstrate the significant superiority of DivBS in the performance-speedup trade-off. The code is publicly available.
Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalization
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Institute of Technology（北京理工大学）；Capital Medical University（首都医科大学）；Tsinghua University（清华大学）
Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality
School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院）；Daniels School of Business, Purdue University（普渡大学丹尼尔斯商学院）；Stern School of Business, New York University（纽约大学斯特恩商学院）
