BPE分词在波兰语中的局限性：屈折语言模型中的分词-屈折形式、语法锚定与第一人称稳定性
The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Models
浏览论文内容
中文总结 AI 辅助
本文以波兰语为案例，揭示BPE分词在屈折语言中无法稳定锚定语法范畴，提出语法形式锚定概念，并建议通过次词汇稳定化、屈折系统锚定及维持语法“我”来改进建模。
中文摘要 AI 辅助
本文以波兰语为测试案例，分析BPE分词在屈折语言中统计分词的局限性。文章探讨基于频率的分词是否能保留语言学相关单位，包括正字法形式、音位与音节切分、派生结构、屈折词尾、语法形式及说话主体。研究材料包括诊断性词汇、儿童文本、波兰共和国宪法序言中的选定形式、词族测试，以及包含波兰语变音符号和鼻元音的示例。BPE分词器可能产生与音节或形态可解释切分相重合的片段，但仍依赖于书面形式的频率。它们不能系统地将正字法表征映射到音位结构或依赖语境的语音实现上。结果表明，BPE稳定的是语法标记的频繁表面片段，而非语法范畴本身。诸如ustanawiamy这样的形式不仅仅是结尾为-y的序列，而是锚定于变位、人称、数、时态、语气和体的动词形式。本文提出了语法形式锚定的概念。在波兰语中，poszlam、zrobilam或bylam等形式无需显式代词即可确立说话主体的位置。在与人工智能的交互中，这暴露了另一个问题：语言模型不具备稳定的语法“我”，而是通过语境重构它，并可能镜像用户的形式或改变语法性别。Roclawski的分词-屈折形式被提议作为评估分词边界的诊断框架。对波兰语更稳定的建模可能需要次词汇稳定化、将语法形式锚定于屈折系统、表征句子模式和动词配价，以及在对话中维持语法“我”。
英文摘要
This article analyzes BPE tokenization in Polish as a test case for the limits of statistical segmentation in an inflectional language. It asks whether frequency-based tokenization preserves linguistically relevant units, including orthographic form, phonemic and syllabic segmentation, derivational structure, inflectional endings, grammatical form, and the speaking subject. The material includes diagnostic words, a children's text, selected forms from the Preamble to the Constitution of the Republic of Poland, word-family tests, and examples with Polish diacritics and nasal vowels. BPE tokenizers may produce segments that coincide with syllabic or morphologically interpretable divisions, but remain dependent on the frequency of written forms. They do not systematically map orthographic representation onto phonemic structure or context-dependent phonetic realization. The results show that BPE stabilizes frequent surface fragments of grammatical exponents rather than grammatical categories themselves. A form such as ustanawiamy is not merely a sequence ending in -y, but a verbal form anchored in conjugation, person, number, tense, mood, and aspect. The article develops the concept of grammatical form anchoring. In Polish, forms such as poszlam, zrobilam, or bylam can establish the position of the speaking subject without an explicit pronoun. In interaction with AI, this exposes a further problem: a language model does not possess a stable grammatical "I", but reconstructs it contextually and may mirror the user's forms or shift grammatical gender. Roclawski's segmentation-flexional forms are proposed as a diagnostic framework for evaluating tokenization boundaries. More stable modeling of Polish may require sublexical stabilization, anchoring grammatical form in the inflectional system, representing sentence patterns and verbal valency, and maintaining the grammatical "I" in dialogue.
发表机构
- University of Lower Silesia DSW（下西里西亚DSW大学）
机构由 AI 辅助整理，请以论文原文为准。
