紧凑语言，复杂模型转变：歧义与欠指定如何及在何处影响大语言模型
Compact Language, Complex Model Shifts: How and Where Ambiguity and Underspecification Affect LLMs
查看机构详情
- HAW Hamburg（汉堡应用科学大学）
- Universität Hamburg（汉堡大学）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
本研究通过人工伪词分析歧义与欠指定对语言模型训练的影响，发现二者提升性能但降低相关序列生成准确性，并揭示内部表示差异。
中文摘要 AI 辅助
我们分析了词汇歧义和欠指定如何影响语言模型的训练。我们创建了人工同音异义词和人工上位词作为伪词，并分析了语言模型在训练过程中随着这些歧义或欠指定伪词类型数量的增加而表现出的生成性能。我们进一步分析了模型是否对歧义或欠指定的陈述进行消歧，并首次从机制上解释了歧义和消歧在内部是如何表示的。我们的主要结果表明，歧义和欠指定都会以与其对语言类型-标记比率影响成比例的方式提高模型性能。然而，生成包含歧义词或其同义词的序列的准确性相对于其他文本有所下降。我们还表明，伪词的内部表示反映了伪同音异义词的消歧，但在生成过程中，伪上位词的欠指定被保留。
英文摘要
We analyze how lexical ambiguity and underspecification affect language model training. We create artificial homonyms and artificial hypernyms as pseudowords and analyze the generative performance of language models as they are trained with increasing amounts of these ambiguous or underspecified pseudoword types. We further analyze whether the models disambiguate ambiguous or underspecified statements and provide a first mechanistic account of how ambiguity and disambiguation are represented internally. Our main results show that both ambiguity and underspecification increase model performance in ways that scale with their influence on the language's type-token ratio. However, the accuracy of generating sequences containing ambiguous words or their synonyms decreases compared to other texts. We also show that internal representations of pseudowords reflect disambiguation of pseudo-homonyms, but underspecification of pseudo-hypernyms is maintained during the generative process.
