LaCy: 小型语言模型能学且应学的不仅仅是损失问题
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
AI总结：
研究在预训练中，小型语言模型（SLM）应学习哪些token以及应通过<CALL>委托哪些token，提出结合损失和事实性信号的LaCy方法，提升SLM在级联生成中的事实准确性。
摘要：
语言模型不断增长以将更多世界知识压缩到其参数中，但可预训练到其中的知识受参数规模上限约束。尤其是小型语言模型（SLM）容量有限，导致事实性错误生成。通常通过让SLM访问外部源（如查询更大模型、文档或数据库）来缓解此问题。在此设置下，我们研究基本问题：预训练期间SLM可以且应该学习哪些token，以及哪些应通过<CALL> token委托。我们发现这不仅仅是损失问题：尽管损失可预测预测token是否与真实值不匹配，但不足以识别哪些预测实际上会导致事实性或语义无效的延续。一些高损失token对应预训练文档中可接受的替代延续，因此不应触发<CALL>。这表明可学习性不能仅从损失表征，而需要关于token在句子中角色的额外领域特定信号。在类似维基百科的领域中，我们展示用spaCy解析器的轻量级语法信息增强损失信号可显著改善委托决策。基于此洞察，我们提出LaCy，一种新颖的预训练方法，结合损失与事实性信号以决定SLM应学习哪些token。实验表明，LaCy模型成功学习预测哪些token以及何时请求帮助。这在与更大模型级联生成时获得更高FactScore，且优于Rho或LLM-judge训练的SLM，同时更简单更廉价。
英文摘要：
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their parameter size. Especially the capacity of Small Language Models (SLMs) is limited, leading to factually incorrect generations. This problem is often mitigated by giving the SLM access to an outside source: the ability to query a larger model, documents, or a database. Under this setting, we study the fundamental question of \emph{which tokens an SLM can and should learn} during pretraining, versus \emph{which ones it should delegate} via a \texttt{<CALL>} token. We find that this is not simply a question of loss: although the loss is predictive of whether a predicted token mismatches the ground-truth, it is insufficient for identifying which predictions would actually lead to factual or semantically invalid continuations. Some high-loss tokens correspond to \emph{acceptable} alternative continuations of a pretraining document and therefore should not trigger a \texttt{<CALL>}. This suggests that learnability cannot be characterized from loss alone, but requires additional domain-specific signals about the role of a token in the sentence. In Wikipedia-like domains, we show that augmenting the loss signal with lightweight grammatical information from a spaCy parser substantially improves delegation decisions. Based on this insight, we propose LaCy, a novel pretraining method that combines loss with factuality signals to decide which tokens an SLM should learn. Our experiments demonstrate that LaCy models successfully learn which tokens to predict and when to call for help. This results in higher FactScores when generating in a cascade with a bigger model and outperforms Rho or LLM-judge trained SLMs, while being simpler and cheaper.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
