Sample weighting is widely used in deep learning. A large number of weighting methods essentially utilize the learning difficulty of training samples to calculate their weights. In this study, this scheme is called difficulty-based weighting. Two important issues arise when explaining this scheme. First, a unified difficulty measure that can be theoretically guaranteed for training samples does not exist. The learning difficulties of the samples are determined by multiple factors including noise level, imbalance degree, margin, and uncertainty. Nevertheless, existing measures only consider a single factor or in part, but not in their entirety. Second, a comprehensive theoretical explanation is lacking with respect to demonstrating why difficulty-based weighting schemes are effective in deep learning. In this study, we theoretically prove that the generalization error of a sample can be used as a universal difficulty measure. Furthermore, we provide formal theoretical justifications on the role of difficulty-based weighting for deep learning, consequently revealing its positive influences on both the optimization dynamics and generalization performance of deep models, which is instructive to existing weighting schemes.
LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training
Department of Computer Science（计算机科学系）；University of Toronto（多伦多大学）；Department of Mathematics（数学系）；The Hospital for Sick Children（圣·玛利亚医院）；Department of Statistics（统计学系）；UHN KITE Research Institute（UHN KITE研究所）；T-CAIREM；Vector Institute（向量研究所）；Institute of Biomedical Engineering（生物医学工程研究所）；Rehabilitation Sciences Institute（康复科学研究所）
A Simple General Approach to Balance Task Difficulty in Multi-Task Learning
Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；Southern University of Science and Technology（南方科技大学）
FOSSIL: Regret-Minimizing Curriculum Learning for Metadata-Free and Low-Data Mpox Diagnosis
Computer Science Division, Gwinnett Technical College, Lawrenceville, GA, USA（基普顿技术学院计算机科学系）；Department of Computer Science, University of Kentucky, Lexington, KY, USA（肯塔基大学计算机科学系）；Department of Biomedical Engineering, Kumoh National Institute of Technology, Gumi, South Korea（Kumoh国立技术学院生物医学工程系）；Department of Biology, Georgia Institute of Technology, Atlanta, GA, USA（佐治亚理工学院生物学系）；Oaro Dermatology Center, Seongnam, Gyeonggi-do, South Korea（Oaro皮肤科中心）
MultiMatch: Multihead Consistency Regularization Matching for Semi-Supervised Text Classification
National University of Science and Technology POLITEHNICA Bucharest（波兰技术大学科学与技术国立大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；NVIDIA（NVIDIA公司）；Renius Technologies（Renius技术公司）
Origins of Universal Machine Learning Force-Field Errors in Multicomponent Materials
Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；Kairos Materials；Shanghai Jiao Tong University（上海交通大学）；Shanghai University（上海大学）；Beijing Institute of Technology（北京理工大学）；Shanghai Institute of Ceramics, Chinese Academy of Sciences（中国科学院上海硅酸盐研究所）
