当婴儿教婴儿时：小数据集上学生知识共享能否优于教师引导的蒸馏？
When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?
AI总结：
针对小数据集上数据高效语言模型预训练问题，提出基于深度互学习的学生模型搜索与双层优化动态加权策略，无需教师模型即可匹配甚至超越教师监督方法的性能。
摘要：
我们提交了BabyLM挑战赛的参赛方案，旨在突破数据高效语言模型预训练的边界。该方法基于深度互学习，引入学生模型搜索以实现多样化初始化。我们通过将加权互学习构建为双层优化问题，解决了平等对待学生的局限性。内循环通过在线蒸馏学习紧凑学生，外循环优化权重以从多样化学生中更好地进行知识蒸馏。这种动态加权策略无需教师模型，降低了计算需求。评估显示，无教师方法可匹配或超越教师监督方法。
英文摘要：
We present our submission to the BabyLM challenge, aiming to push the boundaries of data-efficient language model pretraining. Our method builds upon deep mutual learning, introducing a student model search for diverse initialization. We address the limitation of treating students equally by formulating weighted mutual learning as a bi-level optimization problem. The inner loop learns compact students through online distillation, while the outer loop optimizes weights for better knowledge distillation from diverse students. This dynamic weighting strategy eliminates the need for a teacher model, reducing computational requirements. Our evaluations show that teacher-less methods can match or surpass teacher-supervised approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
