双塔语言模型中的自发偏好
Spontaneous Emerging Preference in Two-tower Language Model
摘要：
基础语言模型不断增长的规模为各种类型的下游任务带来了显著的性能提升。由于基础语言模型的庞大规模所带来的副作用（如部署成本、可用性问题和环境成本）的存在，人们有兴趣探索其他可能的方向，如分而治之的方案。在本文中，我们提出了一个基本问题：语言过程是自然可分的吗？我们研究了这个问题，一个简单的双塔语言模型设置，其中两个语言模型具有相同的配置并排合作训练。在这种情况下，我们发现了自发出现的偏好现象，其中一些令牌始终被一个塔更好地预测，而另一些则被另一个塔预测。无论模型配置和类型如何，这种现象都是定性稳定的，这表明这是自然语言的内在属性。这项研究表明，自然语言的有趣的性质仍然有待发现，这可能有助于未来的发展自然语言处理技术。
英文摘要：
The ever-growing size of the foundation language model has brought significant performance gains in various types of downstream tasks. With the existence of side-effects brought about by the large size of the foundation language model such as deployment cost, availability issues, and environmental cost, there is some interest in exploring other possible directions, such as a divide-and-conquer scheme. In this paper, we are asking a basic question: are language processes naturally dividable? We study this problem with a simple two-tower language model setting, where two language models with identical configurations are trained side-by-side cooperatively. With this setting, we discover the spontaneous emerging preference phenomenon, where some of the tokens are consistently better predicted by one tower while others by another tower. This phenomenon is qualitatively stable, regardless of model configuration and type, suggesting this as an intrinsic property of natural language. This study suggests that interesting properties of natural language are still waiting to be discovered, which may aid the future development of natural language processing techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
