Healthcare providers are increasingly using machine learning to predict patient outcomes to make meaningful interventions. However, despite innovations in this area, deep learning models often struggle to match performance of shallow linear models in predicting these outcomes, making it difficult to leverage such techniques in practice. In this work, motivated by the task of clinical prediction from insurance claims, we present a new technique called Reverse Distillation which pretrains deep models by using high-performing linear models for initialization. We make use of the longitudinal structure of insurance claims datasets to develop Self Attention with Reverse Distillation, or SARD, an architecture that utilizes a combination of contextual embedding, temporal embedding and self-attention mechanisms and most critically is trained via reverse distillation. SARD outperforms state-of-the-art methods on multiple clinical prediction outcomes, with ablation studies revealing that reverse distillation is a primary driver of these improvements. Code is available at https://github.com/clinicalml/omop-learn.
Rethinking Reverse KL as Adaptive Entropy Distillation
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Science and Technology, Hong Kong Metropolitan University(香港都会大学科技学院)
Reverse Distillation: Consistently Scaling Protein Language Model Representations
Department of Computer Science(计算机科学系);Duke University(杜克大学);Center for Computational Biology(计算生物学中心);Flatiron Institute(Flatiron研究所);Depts of Biostats. and Bioinformatics(生物统计学与生物信息学系;细胞生物学系);and Cell Biology
Entropy-Aware On-Policy Distillation of Language Models
IBM Research, San Jose, CA, USA(IBM研究院,旧金山,加州,美国);University of Toronto, Ontario, Canada(多伦多大学,安大略,加拿大);Vector Institute, Ontario, Canada(向量研究所,安大略,加拿大)