Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters. Based on this intuition, we explore the use of exact per-sample Hessian-vector products and gradients to construct optimizers that are self-tuning and hyperparameter-free. Based on a dynamics model of the gradient, we derive a process which leads to a curvature-corrected, noise-adaptive online gradient estimate. The smoothness of our updates makes it more amenable to simple step size selection schemes, which we also base off of our estimates quantities. We prove that our model-based procedure converges in the noisy quadratic setting. Though we do not see similar gains in deep learning tasks, we can match the performance of well-tuned optimizers and ultimately, this is an interesting step for constructing self-tuning optimizers.
X-REFINE: XAI-based RElevance input-Filtering and archItecture fiNe-tuning for channel Estimation
1 University of Paris-Est Créteil (UPEC), LISSI/TincNET, F-94400, Vitry-sur-Seine, France. 4 IMT Nord Europe, Institut Mines T\'el\'ecom, Centre for Digital Systems, F-59653 Villeneuve d’Ascq, France. Email
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China.（计算机科学与工程系，南方科技大学，深圳，中国）；Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.（数学系，南方科技大学，深圳，中国）
Convex--Concave Quadratic Spectral Filtering for Graph Neural Networks
School of Computing and Artificial Intelligence, Guangzhou Xinhua University（广州新华学院计算机与人工智能学院）；School of Statistics and Data Science, Guangdong University of Finance and Economics（广东财经大学统计与数据科学学院）
Faculty of Mathematics and Informatics, Hanoi University of Science and Technology（河内科学技术大学数学与信息学院）；Viettel AI, Viettel Group（Viettel集团Viettel人工智能公司）；Department of Mathematics, National University of Singapore（新加坡国立大学数学系）
Technical University of Munich（慕尼黑工业大学）；Costa Rica Institute of Technology（哥斯达黎加理工学院）
When signals are measured through physical sensors, they are perturbed by noise. To reduce noise, low-pass filters are commonly employed in order to attenuate high frequency components in the incoming signal, regardless if they come from noise or the actual signal. Therefore, low-pass filters must be carefully tuned in order to avoid significant deterioration of the signal. This tuning requires prior knowledge about the signal, which is often not available in applications such as reinforcement learning or learning-based control. In order to overcome this limitation, we propose an adaptive low-…
Algorithm selection and hyperparameter tuning remain two of the most challenging tasks in machine learning. Automated machine learning (AutoML) seeks to automate these tasks to enable widespread use of machine learning by non-experts. This paper introduces OBOE, a collaborative filtering method for time-constrained model selection and hyperparameter tuning. OBOE forms a matrix of the cross-validated errors of a large number of supervised learning models (algorithms together with hyperparameters) on a large number of datasets, and fits a low rank model to learn the low-dimensional feature vecto…
The quality of an induced model by a learning algorithm is dependent on the quality of the training data and the hyper-parameters supplied to the learning algorithm. Prior work has shown that improving the quality of the training data (i.e., by removing low quality instances) or tuning the learning algorithm hyper-parameters can significantly improve the quality of an induced model. A comparison of the two methods is lacking though. In this paper, we estimate and compare the potential benefits of filtering and hyper-parameter optimization. Estimating the potential benefit gives an overly optim…
Multilingual Safety Signals Are Multi-Layered: Filtering Safety-Degrading Data for Safer LLMs
Beihang University（北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Qingdao Research Institute, Beihang University（北京航空航天大学青岛研究院）
Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
Yale university（耶鲁大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Hong Kong University of Science（香港科学大学）；Xi’an Jiaotong University（西安交通大学）；Central University of Finance（中央财经大学）
