A Diversity-Enhanced Knowledge Distillation Model for Practical Math Word Problem Solving
Faculty of Artificial Intelligence in Education, Central China Normal University（华中师范大学教育人工智能学院）；School of Information and Safety Engineering, Zhongnan University of Economics and Law（中南财经政法大学信息与安全工程学院）；School of Computer, Central China Normal University（华中师范大学计算机学院）；Information Retrieval and Knowledge Management Research Lab, York University（约克大学信息检索与知识管理研究实验室）
Yi Zhang, Guangyou Zhou, Zhiwen Xie, Jinjin Ma, Jimmy Xiangji Huang
Math Word Problem (MWP) solving is a critical task in natural language processing, has garnered significant research interest in recent years. Various recent studies heavily rely on Seq2Seq models and their extensions (e.g., Seq2Tree and Graph2Tree) to generate mathematical equations. While effective, these models struggle to generate diverse but counterpart solution equations, limiting their generalization across various math problem scenarios. In this paper, we introduce a novel Diversity-enhanced Knowledge Distillation (DivKD) model for practical MWP solving. Our approach proposes an adaptive diversity distillation method, in which a student model learns diverse equations by selectively transferring high-quality knowledge from a teacher model. Additionally, we design a diversity prior-enhanced student model to better capture the diversity distribution of equations by incorporating a conditional variational auto-encoder. Extensive experiments on {four} MWP benchmark datasets demonstrate that our approach achieves higher answer accuracy than strong baselines while maintaining high efficiency for practical applications.
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Google（谷歌）；University of Washington（华盛顿大学）
Knowledge distillation is one of the primary methods of transferring knowledge from large to small models. However, it requires massive task-specific data, which may not be plausible in many real-world applications. Data augmentation methods such as representation interpolation, token replacement, or augmentation with models are applied to tackle this problem. However, these data augmentation methods either potentially cause shifts in decision boundaries (representation interpolation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation wit…
Generative models are increasingly able to produce remarkably high quality images and text. The community has developed numerous evaluation metrics for comparing generative models. However, these metrics do not effectively quantify data diversity. We develop a new diversity metric that can readily be applied to data, both synthetic and natural, of any type. Our method employs random network distillation, a technique introduced in reinforcement learning. We validate and deploy this metric on both images and text. We further explore diversity in few-shot image generation, a setting which was pre…
Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Science and Technology of China（中国科学技术大学）；Shanghai University of Finance and Economics（上海财经大学）
Microsoft Research Asia（微软亚洲研究院）；Zhejiang University（浙江大学）；Peking University（北京大学）
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the…
Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
University of Georgia（佐治亚大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Harvard University（哈佛大学）；Carnegie Mellon University（卡内基梅隆大学）；Vanderbilt University（范德堡大学）；Mayo Clinic Arizona（亚利桑那梅奥诊所）；Augusta University（奥古斯塔大学）
