缩放与指令微调在语言感知中的作用：模型与人类注意力对比
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
AI总结：
本文通过对比不同规模LLaMA、Alpaca和Vicuna的自注意力与人类眼跳，发现缩放能提升模型注意力的人类相似性和有效性，而指令微调主要增强指令敏感性，且当前LLMs注意力更接近非母语者，揭示其语言感知仍非最优。
摘要：
近期的大型语言模型（LLMs）展现出强大的自然语言理解能力。由于它们大多共享相同的基本结构，即 transformer 块，训练过程中可能促成其成功的因素包括缩放（scaling）和指令微调（instruction tuning）。然而，这些因素如何影响模型的语言感知尚不清楚。本研究比较了多个现有 LLMs（LLaMA、Alpaca 和 Vicuna）在不同规模（7B、13B、30B、65B）下的自注意力，并结合眼跳（eye saccade）这一人类阅读注意力的一个方面，以评估缩放和指令微调对语言感知的影响。结果表明，缩放通过减少对琐碎模式的依赖，增强了与人类的相似性并提高了有效注意力，而指令微调并未做到这一点。然而，指令微调显著增强了模型对指令的敏感性。我们还发现，当前的 LLMs 在注意力上始终更接近非母语者而非母语者，这表明所有模型的语言感知都未达到最优。我们在分析中使用的代码和数据可在 GitHub 上获取。
英文摘要：
Recent large language models (LLMs) have revealed strong abilities to understand natural language. Since most of them share the same basic structure, i.e. the transformer block, possible contributors to their success in the training process are scaling and instruction tuning. However, how these factors affect the models' language perception is unclear. This work compares the self-attention of several existing LLMs (LLaMA, Alpaca and Vicuna) in different sizes (7B, 13B, 30B, 65B), together with eye saccade, an aspect of human reading attention, to assess the effect of scaling and instruction tuning on language perception. Results show that scaling enhances the human resemblance and improves the effective attention by reducing the trivial pattern reliance, while instruction tuning does not. However, instruction tuning significantly enhances the models' sensitivity to instructions. We also find that current LLMs are consistently closer to non-native than native speakers in attention, suggesting a sub-optimal language perception of all models. Our code and data used in the analysis is available on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
