Knowledge distillation offers a transformative pathway to developing powerful, yet efficient, small language models (SLMs) suitable for resource-constrained environments. In this paper, we benchmark the performance and computational cost of distilled models against their vanilla and proprietary counterparts, providing a quantitative analysis of their efficiency. Our results demonstrate that distillation creates a superior performance-tocompute curve. We find that creating a distilled 8B model is over 2,000 times more compute-efficient than training its vanilla counterpart, while achieving reasoning capabilities on par with, or even exceeding, standard models ten times its size. These findings validate distillation not just as a compression technique, but as a primary strategy for building state-of-the-art, accessible AI
Microsoft Egypt Development Center（微软埃及开发中心）；Microsoft Corporation（微软公司）
This paper describes our submission to the constrained track of WMT21 shared news translation task. We focus on the three relatively low resource language pairs Bengali to and from Hindi, English to and from Hausa, and Xhosa to and from Zulu. To overcome the limitation of relatively low parallel data we train a multilingual model using a multitask objective employing both parallel and monolingual data. In addition, we augment the data using back translation. We also train a bilingual model incorporating back translation and knowledge distillation then combine the two models using sequence-to-s…
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）；Amazon Alexa（亚马逊Alexa）；Harvard University（哈佛大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Language models excel at generating coherent text, and model compression techniques such as knowledge distillation have enabled their use in resource-constrained settings. However, these models can be biased in multiple ways, including the unfounded association of male and female genders with gender-neutral professions. Therefore, knowledge distillation without any fairness constraints may preserve or exaggerate the teacher model's biases onto the distilled model. To this end, we present a novel approach to mitigate gender disparity in text generation by learning a fair model during knowledge …
Research Institute for Artificial Intelligence, Romanian Academy（罗马尼亚科学院人工智能研究所）
Running large-scale pre-trained language models in computationally constrained environments remains a challenging problem yet to be addressed, while transfer learning from these models has become prevalent in Natural Language Processing tasks. Several solutions, including knowledge distillation, network quantization, or network pruning have been previously proposed; however, these approaches focus mostly on the English language, thus widening the gap when considering low-resource languages. In this work, we introduce three light and fast versions of distilled BERT models for the Romanian langu…
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge di…
Videos shot by laymen using hand-held cameras contain undesirable shaky motion. Estimating the global motion between successive frames, in a manner not influenced by moving objects, is central to many video stabilization techniques, but poses significant challenges. A large body of work uses 2D affine transformations or homography for the global motion. However, in this work, we introduce a more general representation scheme, which adapts any existing optical flow network to ignore the moving objects and obtain a spatially smooth approximation of the global motion between video frames. We achi…
MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
Seoul National University（首尔大学）；Seoul National University Hospital（首尔大学医院）；Seoul National University College of Medicine（首尔大学医学院）；Sungkyunkwan University School of Medicine（成均馆大学医学院）
Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models
Zhejiang University（浙江大学）；School of Public Health, Zhejiang University（浙江大学公共卫生学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Liangzhu Laboratory and Institute of Wenzhou, Zhejiang University（浙江大学良渚实验室与温州研究院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Benchmarking Resource-Efficient LLMs for Research Topic Ontology Generation in the Biomedical Field
Knowledge Media Institute, The Open University, Milton Keynes, UK（开放大学知识媒体研究所）；Department of Business and Law, University of Milano-Bicocca, Milan, IT（米兰-比科卡大学商业与法律系）
