Mini Minds：探索 Bebeshka 与 Zlata Baby 模型
Mini Minds: Exploring Bebeshka and Zlata Baby Models
AI总结：
本文提出面向 BabyLM 竞赛 Strict-Small 赛道的两个小规模语言模型 Bebeshka 和 Zlata，通过架构搜索优化掩码语言建模，在仅约一半基线规模下取得可比性能，并探索其在道德判断任务中与人类价值观对齐的潜力。
摘要：
在本文中，我们描述了里昂第二大学提交给 BabyLM 竞赛 Strict-Small 赛道的参赛系统。该共享任务的创建重点是强调在有限规模数据上从零开始进行小规模语言建模以及人类语言习得。为 Strict-Small 赛道发布的数据集包含 1000 万个词，这与儿童的词汇量规模相当。我们通过架构搜索来处理该任务，在共享任务的数据上最小化掩码语言建模损失。在找到最优配置后，我们引入了两个提交评估的小规模语言模型（LMs）：一个具有 8 个注意力头的 4 层编码器和一个具有 12 个头的 6 层解码器模型，我们分别将其命名为 Bebeshka 和 Zlata。尽管规模仅为基线语言模型的一半，我们提出的模型仍取得了可比的性能。我们进一步探索了小规模语言模型在涉及道德判断的任务中的适用性，使其预测与人类价值观保持一致。这些发现凸显了紧凑型语言模型在解决实际语言理解任务方面的潜力。
英文摘要：
In this paper, we describe the University of Lyon 2 submission to the Strict-Small track of the BabyLM competition. The shared task is created with an emphasis on small-scale language modelling from scratch on limited-size data and human language acquisition. Dataset released for the Strict-Small track has 10M words, which is comparable to children's vocabulary size. We approach the task with an architecture search, minimizing masked language modelling loss on the data of the shared task. Having found an optimal configuration, we introduce two small-size language models (LMs) that were submitted for evaluation, a 4-layer encoder with 8 attention heads and a 6-layer decoder model with 12 heads which we term Bebeshka and Zlata, respectively. Despite being half the scale of the baseline LMs, our proposed models achieve comparable performance. We further explore the applicability of small-scale language models in tasks involving moral judgment, aligning their predictions with human values. These findings highlight the potential of compact LMs in addressing practical language understanding tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
