小型语言模型能否学习、遗忘并保留噪声模式？
Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?
AI总结：
本研究探究10亿至30亿参数的小型语言模型学习、保留和消除噪声的能力，发现模型大小和预训练数据质量影响噪声处理，并提出了通过干净数据训练减轻噪声影响的策略。
摘要：
随着在资源受限环境中对高效语言模型的需求日益增长，小型语言模型（SLMs）已成为大型语言模型（LLMs）的紧凑且实用的替代方案。尽管已有研究探讨了LLMs中的噪声处理，但关于SLMs如何处理噪声（其可靠实际部署的关键因素）知之甚少。本研究调查了参数在10亿至30亿之间的SLMs学习、保留并随后消除不同类型噪声（词翻转、字符翻转、音译、无关内容和矛盾信息）的能力。四个预训练的SLMs（Olmo 1B、Qwen1.5 1.8B、Gemma1.1 2B和Phi2 2.7B）在无噪声数据上进行指令微调，并使用上下文示例进行测试以评估噪声学习。随后，在指令微调中引入噪声模式以评估其适应性。结果显示模型处理噪声的方式存在差异，较小的模型如Olmo能快速适应噪声模式。Phi2精心策划、结构化且高质量的预训练数据使其能够抵抗字符级、音译和反事实噪声，而Gemma通过其多语言预训练成功适应了音译噪声。随后的干净数据训练有效减轻了噪声影响。这些发现为开发用于实际应用的稳健SLMs提供了实用策略。
英文摘要：
With the growing need for efficient language models in resource-constrained environments, Small Language Models (SLMs) have emerged as compact and practical alternatives to Large Language Models (LLMs). While studies have explored noise handling in LLMs, little is known about how SLMs handle noise, a critical factor for their reliable real-world deployment. This study investigates the ability of SLMs with parameters between 1 and 3 billion to learn, retain, and subsequently eliminate different types of noise (word flip, character flip, transliteration, irrelevant content, and contradictory information). Four pretrained SLMs (Olmo 1B, Qwen1.5 1.8B, Gemma1.1 2B, and Phi2 2.7B) were instruction-tuned on noise-free data and tested with in-context examples to assess noise learning. Subsequently, noise patterns were introduced in instruction tuning to assess their adaptability. The results revealed differences in how models handle noise, with smaller models like Olmo quickly adapting to noise patterns. Phi2's carefully curated, structured, and high-quality pretraining data enabled resistance to character level, transliteration, and counterfactual noise, while Gemma adapted successfully to transliteration noise through its multilingual pretraining. Subsequent clean data training effectively mitigated noise effects. These findings provide practical strategies for developing robust SLMs for real-world applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
