一个每一层都重要的惊奇预言机
A surprisal oracle for when every layer counts
AI总结：
针对BabyLM 2024任务，提出更新的主动课程语言建模（ACLM）流程，改进相似性模型以增强动态性，在ELC-BERT上运行后，虽细粒度语法推理表现不佳，但常识和世界知识任务优于官方基线。
摘要：
主动课程语言建模（ACLM；Hong等人，2023）是一种由学习者主导的语言模型训练方法。我们在提交给BabyLM 2023任务的成果中提出了该流程的原始版本，现在为BabyLM 2024任务提出了更新后的ACLM流程。ACLM包含一个通过不确定性模型在训练过程中迭代且动态构建的课程；与最不确定候选项目具有相似不确定性的其他训练项目会被优先考虑。我们的新流程改进了相似性模型，使其更具动态性，并在BabyLM 2023任务中最成功的模型ELC-BERT（Charpentier和Samuel，2023）上运行ACLM。我们发现，虽然我们的模型在细粒度语法推理上表现不佳，但在常识和世界知识任务上优于BabyLM 2024的官方基线。我们的代码可在https://github.com/asayeed/ActiveBaby获取。
英文摘要：
Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
