Morpheus: 一种面向土耳其语的形态感知神经分词器和词嵌入器
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
AI总结：
针对土耳其语粘着特性，提出Morpheus神经词素边界模型，实现无损可逆分词与结构化词嵌入，在可逆分词器中达到最低比特每字符（1.425），词素对齐F1提升至0.61，GPU内存节省约19%。
摘要：
土耳其语是粘着语：意义由词素承载，然而驱动现代语言模型的子词分词器根据语料库统计分割单词，切碎了承载语义的后缀，并且在WordPiece和基于规则的分析器的情况下，无法将其输出解码回原始文本。本文提出\textbf{Morpheus}，一个面向土耳其语的神经词素边界模型，它同时是一个无损的、形态感知的分词器和一个词嵌入生成器。一个可微的泊松-二项式动态规划程序在训练期间将每个字符的边界概率转化为软词素隶属度，在推理时转化为精确的片段，无需字符串归一化，因此$\mathrm{decode}(\mathrm{encode}(w)) = w$由构造保证。由于该模型是神经模型，相同的正向传播在分词的同时也输出结构化的词嵌入。在可逆分词器中——唯一适用于生成的分词器——Morpheus达到了最低的比特每字符（1.425），将子词家族的金标准词素对齐大致翻倍（MorphScore宏F1从约0.32提升至0.61），并且相比64K词汇量的子词分词器节省了约19%的GPU内存。作为嵌入器，冻结的Morpheus向量在词汇检索（根家族MAP 0.85）和同根验证（ROC-AUC 1.00）上领先，超越了多语言检索器BGE-M3和BERTurk；在上下文和屈折依赖的任务（NER、格/数探测）上，更重的上下文编码器仍然领先——我们将这一权衡归因于Morpheus以词根为中心的几何结构。代码：此https URL 模型：此https URL 交互演示：此https URL。
英文摘要：
Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents \textbf{Morpheus}, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so $\mathrm{decode}(\mathrm{encode}(w)) = w$ holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character ($1.425$), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 $0.61$ vs.\ ${\sim}0.32$), and uses ${\sim}19\%$ less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP $0.85$) and same-root verification (ROC-AUC $1.00$), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
