One of the biggest bottlenecks in a machine learning workflow is waiting for models to train. Depending on the available computing resources, it can take days to weeks to train a neural network on a large dataset with many classes such as ImageNet. For researchers experimenting with new algorithmic approaches, this is impractically time consuming and costly. We aim to generate smaller "proxy datasets" where experiments are cheaper to run but results are highly correlated with experimental results on the full dataset. We generate these proxy datasets using by randomly sampling from examples or classes, training on only the easiest or hardest examples and training on synthetic examples generated by "data distillation". We compare these techniques to the more widely used baseline of training on the full dataset for fewer epochs. For each proxying strategy, we estimate three measures of "proxy quality": how much of the variance in experimental results on the full dataset can be explained by experimental results on the proxy dataset.
Experiments on Imagenette and Imagewoof (Howard, 2019) show that running hyperparameter search on the easiest 10% of examples explains 81% of the variance in experiment results on the target task, and using the easiest 50% of examples can explain 95% of the variance, significantly more than training on all the data for fewer epochs, a more widely used baseline. These "easy" proxies are higher quality than training on the full dataset for a reduced number of epochs (but equivalent computational cost), and, unexpectedly, higher quality than proxies constructed from the hardest examples. Without access to a trained model, researchers can improve proxy quality by restricting the subset to fewer classes; proxies built on half the classes are higher quality than those with an equivalent number of examples spread across all classes.
A Proxy Attack-Free Strategy for Practically Improving the Poisoning Efficiency in Backdoor Attacks
Nanjing University of Information Science and Technology（南京信息工程大学）；University of Science and Technology of China（中国科学技术大学）；Huawei Technologies Co. Ltd.（华为技术有限公司）；Huazhong University of Science and Technology（华中科技大学）；China Academic of Electronics and Information Technology（中国电子信息产业发展研究院）
EcoNAS: Finding Proxies for Economical Neural Architecture Search
The University of Sydney（悉尼大学）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）
神经结构搜索（NAS）在许多计算机视觉任务中取得了重大进展。虽然已经提出了许多方法来提高NAS的效率，但是搜索过程仍然是费力的，因为在大的搜索空间上训练和评估合理的架构是耗时的。在代理下评估网络候选（即，计算上减少的设置）因此变得不可避免。在本文中，我们观察到，大多数现有的代理表现出不同的行为，在保持网络候选人之间的排名一致性。特别是，一些代理可以更可靠-候选人的排名没有太大的差异，比较其减少设置性能和最终性能。在本文中，我们系统地研究了一些广泛采用的约化因子，并报告了我们的观察结果。受这些观察的启发，我们提出了一个可靠的代理，并进一步制定了分层代理策略。该策略将更多的计算花费在可能更准确的候选网络上，同时使用快速代理在早期丢弃不太有希望的网络。这就产生了一种经济的基于进化的NAS（EcoNAS），与基于进化的最新技术相比，它实现了令人印象深刻的400倍搜索时间减少（8 vs. 3150 GPU天）。由我们的观察结果领导的一些新代理也可以应用于加速其他NAS方法，同时仍然能够发现性能与以前的代理策略相匹配的良好候选网络。
Zero-Shot Neural Architecture Search with Weighted Response Correlation
School of Internet, Anhui University（安徽大学互联网学院）；Alibaba Group Holding Limited（阿里巴巴集团控股有限公司）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Alibaba International Digital Commerce（阿里巴巴国际数字商业）；Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳高性能数据挖掘重点实验室，深圳先进技术研究所，中国科学院）
Building high accuracy emulators for scientific simulations with deep neural architecture search
University of Oxford（牛津大学）；University of Rochester（罗切斯特大学）；Yale University（耶鲁大学）；University of Seville（塞维利亚大学）；Max-Planck-Institut für Plasmaphysik（马克斯·普朗克等离子体物理研究所）
Combined Pruning for Nested Cross-Validation to Accelerate Automated Hyperparameter Optimization for Embedded Feature Selection in High-Dimensional Data with Very Small Sample Sizes
Ostfalia University of Applied Sciences（东法利亚应用科学大学）；Clausthal University of Technology（克劳斯塔尔工业大学）；Helmholtz Centre for Infection Research（亥姆霍兹感染研究中心）
Department of Operations Research and Financial Engineering（运营研究与金融工程系）；School of Engineering and Applied Science（工程与应用科学学院）；Princeton University（普林斯顿大学）
Efficient Hyperparameter Optimization for LLM Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；China United Network Communications Group（中国联合网络通信集团）
Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
John Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·保罗森工程与应用科学学校）；Center for Brain Sciences（脑科学研究中心）；Kempner Institute（凯普纳研究所）
Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking
Basque Center for Applied Mathematics（巴斯克应用数学中心）；IKERBASQUE - Basque Foundation for Science（伊克巴斯克-巴斯克科学基金会）；Columbia University（哥伦比亚大学）；Walmart Global Tech（沃尔玛全球技术）；eBay Inc.（易贝公司）
From Entity Reliability to Clean Feedback: An Entity-Aware Denoising Framework Beyond Interaction-Level Signals
University of Science；Technology of China \& State Key Laboratory of Cognitive Intelligence China；Technology of China \& State Key Laboratory of Cognitive Intelligence
DeepStock: Reinforcement Learning with Policy Regularizations for Inventory Management
Booth School of Business, University of Chicago（芝加哥大学商学院）；Taobao & Tmall Group（淘宝与天猫集团）；School of Economics, Sichuan University（四川大学经济学院）；Graduate School of Business, Columbia University（哥伦比亚大学商学院）；School of Operations Research and Information Engineering, Cornell University（康奈尔大学运筹学与信息工程学院）
